$\alpha$-ReQ : Assessing Representation Quality in Self-Supervised Learning by measuring eigenspectrum decay
Authors: Kumar Krishna Agrawal
Arnab Kumar Mondal
Arna Ghosh
Blake Aaron Richards
Venue: NeurIPS 2022 Accept (2022)
Keywords: Representation analysis
Representation learning
Self Supervised Learning
Eigenspectrum
Overparameterized Linear regression
Computer vision
Model Selection

Abstract: Self-Supervised Learning (SSL) with large-scale unlabelled datasets enables learning useful representations for multiple downstream tasks. However, assessing the quality of such representations efficiently poses nontrivial challenges. Existing approaches train linear probes (with frozen features) to evaluate performance on a given task. This is expensive both computationally, since it requires retraining a new prediction head for each downstream task, and statistically, requires task-specific labels for multiple tasks. This poses a natural question, how do we efficiently determine the "goodness" of representations learned with SSL across a wide range of potential downstream tasks? In particular, a task-agnostic statistical measure of representation quality, that predicts generalization without explicit downstream task evaluation, would be highly desirable. 
   
In this work, we analyze characteristics of learned representations $\mathbf{f_\theta}$, in well-trained neural networks with canonical architectures \& across SSL objectives. We observe that the eigenspectrum of the empirical feature covariance $\mathrm{Cov}(\mathbf{f_\theta}$) can be well approximated with the family of power-law distribution. We analytically and empirically (using multiple datasets, e.g. CIFAR, STL10, MIT67, ImageNet) demonstrate that the decay coefficient $\alpha$ serves as a measure of representation quality for tasks that are solvable with a linear readout, i.e. there exist well-defined intervals for $\alpha$ where models exhibit excellent downstream generalization. Furthermore, our experiments suggest that key design parameters in SSL algorithms, such as BarlowTwins, implicitly modulate the decay coefficient of the eigenspectrum ($\alpha$). As $\alpha$ depends only on the features themselves, this measure for model selection with hyperparameter tuning for BarlowTwins enables search with less compute.

a-ReQ Assessing representation quality by measuring eigenspectrum decay
Kumar Krishna Agrawalt UC Berkeley CA, USA
Arnab Kumar Mondal Mila & McGill University Montréal, QC, Canada
Arna Ghosh
Mila & McGill University Montréal, QC, Canada
Blake A. Richards
Mila, Montreal Neurological Institute & McGill University Montréal, QC, Canada
Learning in Machines and Brains Program, CIFAR Toronto, ON, Canada
Abstract
Supervupe Learning (SSL) with large-scale unlabelled datasets enables learn- ing useful representations for multiple downstream tasks. However, efficiently assessing the quality of such representations poses nontrivial challenges. Existing approaches train linear probes (with frozen features) to evaluate performance given task. This is expensive both computationally. since it requires retraining a new prediction head for each downstream task, and statistically, which requires task-specific labels for multiple tasks. This poses a natural question, how do we efficiently determine the "goodness" of representations learned with SSL across a wide range of potential downstream tasks? In particular, a task-agnostic statisti- cal measure representation quality that predicts generalization without explicit downstream task evaluation would be highly desirable.
In this work, analyze characteristics of learned representations fg in well-trained neural networks with canonical architectures & across SSL objectives. We ob- that the eigenspectrum of the empirical feature covariance Cov(fe) be well approximated with the family of a power-law distribution. We analytically and empirically (using multiple datasets, e.g. CIFAR. STL10, MIT67, ImageNet) demonstrate that the decay coefficient a serves as a measure of representation quality for tasks that solvable with linear readout, that there exist well- defined intervals for a where models exhibit excellent downstream generalization. Furthermore, experiments suggest that key design parameters in SSL algo- rithms, such as HarlowTwing [1], implicitly modulate the decay coefficient of the eigenspectrum (a). As depends only the features themselves, this measure for model selection with hyperparameter tuning for BarlowTwin enables the search with less compute.
Introduction
The recent success self-supervised learning (SSL) has changed the landscape of deep learning significantly. With well-engineered architectures and training objectives, SSL models learn useful representations from large datasets without relying on any labels [1, 2, 3]. Despite this progress, quantifying the representation quality for models trained with SSL is still open problem. The
These authors contributed equally to this work
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

A
8
B
Cest
2.07, R² 00.822
observation regression
eigenvalue (log)
Deep Neural Network
2
Coy,00
eigenvalue index (log)
Figure 1: (A) One approach evaluate representation quality is tracking the eigenspace of feature co- variance matrix E, (f) = 1/n Σ.=1 f(x))((x;)T Analysing populations of neural activation [6. 7]. suggests that the eigenspectrum for En, can be approximated by power-law, where A о i= (B)
For a ResNet-50 model pretrained ImageNet, extract activations from intermediate layer. We plot the eigenvalue spectrum {AI.... Ad} for covariance matrix Σ, (resnet50 (feats= (feats='blocke) bblock4)))
in log-log scale. Further, fitting linear regressor we estimate the decay cocfficient á.
most obvious (and common) solution is to assess performance of models using these representations on downstream tasks, However, if the goal is general representations that can be used across many domains, then this either requires a large investment of time and energy to be done well (in order many tasks and datasets). A Alternatively, if models are assessed on only small number of datasets and tasks, then it is hard to be confident in the assessment. Thus, we are left with question: Can we assess the quality of learned representations without explicitly evaluating the performance on downstream tasks? To answer this question, we must formally define the "quality" of representations and subsequently examine statistical estimators that measure this property without needing downstream evaluation. Beyond theoretical interest, such a metric would be highly desirable for model selection and also useful for designing new SSL algorithms.
to assess performance
In search of such metric, we turn our attention to one of the more efficient learning machines in existence the mammalian brain. The hierarchical and distributed organization of neural circuits, especially in the cortex, provides neural representations that support a wide array of behaviours across many domains. For example, representations in primary visual cortex (VI) of mammalian brains are used by animals to support downstream behaviours ranging from object categorization to movement detection and motor control [4, 5J. This berth of downstream uses of V1 representations is desirable for artificial vision systems trained with SSL. Thus, understanding the properties of representations in VI is reasonable starting point for seeking a general metric of representation quality.
Recent breakthroughs in systems neuroscience enable large-scale recordings of neural activity. By recording and analyzing the response to visual stimuli, [6, 7] find that activations in the mouse and macaque monkey V1 exhibit a characteristic information geometric structure. In particular, these representations are high-dimensional, yet the amount of information encoded along the different principal directions varies significantly. Notably, this variance (computed by measuring the eigen- spectrum of the empirical covariance matrix) is apll-appro by power-law distribution with decay coefficient & 1, i.e., the th eigenvalue of the covariance matrix scales as 1/n.
Motivated by these results, we explore the use of the decay rate of the empirical eigenspectrum to characterize representation quality in neural networks trained with SSL (Fig. la). Across diverse model architectures, pretraining objectives, and downstream classification tasks, we empirically observe an eigenspectrum decay in the representation covariance matrix that roughly follows power-law distribution (see e.g. Fig. 1b). We also find that the coefficient of this decay, denoted by a, informative of downstream generalization performance. Importantly, a can efficiently be calculated without any labels, and thereby, could be incorporated into existing SSL pipelines for efficient model selection on fixed compute budget. Our core contributions this paper are:
1. a as potential metric We observe that canonical pretrained architectures have representa- tions which loosely conform to power-law distribution in their covariance eigenspectrum. Under an assumption of a power-law distribution, we prove that the convergence rate and upper bound on generalization error are related to decay-coefficient (a) of the corresponding plwer-law distribution, with best values of being neither too large nor too small.

---

2. a as label-free measure for representation quality: We empirically validate our theoret- ical results by demonstrating a relationship between and downstream generalization. In particular, we find that either too high or too low an a value implies poor generalization, both in-distribution and out-of-distribution. Generally, the best representations are those where a is in a range that is close to 1, as observed in V1 of the real brain. Furthermore, these results hold irrespective of the choice of network architecture or pretraining objective.
3. a for model selection SSL: We establish a as a reliable metric for model selection in a specific case of SSL, Barlow Twins [1]. Notably, we show that allows to identify model hyper- parameters that lead to representations that generalize well without any labels, more so than the actual loss function used to train the network.
Altogether, results show that is a promising task/architecctur/ddaa agnostic metric for assessing representation quality in SSL. We publicly release our results and code
2 Theoretical Framework to Assess Representations in SSL
We are interested in evaluating the quality of high dimensional representations learned by neural networks. Formally, consider the overparameterizee setting, with datasets X' C R", and learned mappings f: Rd + IR such that f(x) is a vector of D-dimensional features.
We consider DNNs as our function approximators, where each architecture implicitly defines function class J {fg 0 E )) where C R2 is the feasible set of model parameters (e-g bounded [-B, B| for some B E R... The search for good representations poses the following optimization problem: with a dataset Prretrain from some data distribution IP Prretrain (potentially with labels), search for optimal parameters 0* such that with pretrain objective Lpretrain (f, D)
0" arg min Spretroin (fg, Dpretrain) FTE
The above optimization problem is usually non-convex, and often uses gradient based optimizer to find an approximate solution 0. Typically, to measure the quality of fa- researchers evaluate the quality of representations on a downstream task Daownstream with linear readout using some metric R(1inear((g), Ddoonstream). A concrete example is pretraining VGG-16 model (F) ImageNet dataset (D preetaiin) and evaluating the learned representations using classification accuracy (R) by learning linear classifier on the MIT67 dataset (Daownstream). For the rest of the paper, we denote feature maps as (2) E RDP where fo X' + RR, and the readout network as go : IR - Rt where k is the target dimensionality. For simplicity of analysis, consider linear readouts unless explicitly mentioned, i.c. go(x) x²
2.1 Covariance estimation and eigenspectrum
For parameterized function fo : X' - R D (assume centered), the (n-sample) empirical covariance matrix is defined as:
(G)
(o(r)))o(x.))
The eigenspectrum of Σ,, (fo) informs us about the variance explained by each principal component of the space spanned by representations fo (x).. Using the spectral decomposition theorem on symmetric matrices, U UUT,, where A is diagonal matrix with nonnegative entries, and U is a matrix
whose columns are the eigenvectors of Σ. Without loss of generality we assume that X 2 X2... 2 Am. where m min(n, D) is the rank of E,, (fo).
2.2 Eigenspectrum Decay in Deep Representation Learning
Recent work in characterizing representation structure in canonical DNNs has demonstrated that the covariance eigenspectrum roughly follows a power-law [8, 9, 10]. Specifically, the eigenspectrum
3

---

of a covariance matrix follows a power-law distribution PL(a), or zeta distribution if for X3 E [Amins Amarl, the eigenvalues Xy are all nonnegative, and
X x j
for some 0. Here, a is the slope of the power law, and is referred as the coefficient of decay of the eigenspectrum. Intuitively, small a (typically a S 1) suggests a dense encoding, while a high a (rapid decay) corresponds to sparse encoding.
Linear Regression Recent work in theoretical machine learning has connected bounds on generalization error for a linear regression problem to the eigenspectrum of the feature covariance matrix. Specifically, in the infnite-ddmensionall setting with + oc, [1]] studied the linear regression setting with Gaussian features, and proved that if the eigenspectrum follows a power-law distribution (up to polylogarithm factors), the min- min-norm solution provides good generalization performance iff a 1. The asymptotic regime of infinite width is excellent framework to study theoretical properties of DNN representations. However, despite having a large number of parameters, practical DNNs always possess finite dimensional representations, making it important to investigate the implications of such results in finite width models. In particular, for the finite dimensional setting we try to answer: Does a sufficiently larger or smaller than still allow efficient learning and strong generalizability?
Insights from
To answer this question, we narrow our focus to gradient-based based optimization techniques usually used to train DNNs. In particular, from the optimization perspective, Advani et al. showed that for deep linear regression in high dimensions, the time required for training and the steady-state generalization error are both O Amein [12]. A key difference from our work is that they assume the inputs are drawn from an isotropic Gaussian distribution. Instead, we investigate the generalization of linear regression on fo (z), which has power law structure in its covariates. We show that the training convergence time grows exponentially with a using the following theorem:
Theorem 2.1. Let ò = fo(x)Ty vp be an overparameterized linear regression problem where tb is learned using gradient descent in order to optimize the training error, Er,gl(y (z)) V),, where (a. y) ~ Drrain. If we assume a power-law distribution in the eigenspectrum of representations at fo, i.e. Xm. minimize the training error, T Teonvergence = O(Na) where number of training samples.
Vn 2 n², where n² 21,2...N }, then the time required by gradient descent to
The outline of the proof builds on key results relating to gradient descent dynamics from [13]. We show that gradient descent updates, when i is initialized to O, yield a recursive relation for (kk), i.e. d after k update steps. Plugging this relation in the gradient formulation, length along the th principal direction of (z() shrinks exponentially with a decay rate proportional to A, Therefore, the time to convergence in training is controlled by the smallest eigenvalue which, by design, follows the power law. In sum, Theorem 2.1 provides an explanation against arbitrarily large values of a.
show that the update step
Next we show that a can't be arbitrarily small as the upper bound on generalization error is higher for smaller values of a.
Theorem 2.2. Let ij = (z) be a linear regression problem as before. Let us further assume that (x) Vx Dirain is representative subset of the inputs from true data distribution: (x, 3) D. Assuming a power-law distribution in the eigenspectrum of representations at fg, i.e. An n n", where 2[1,2...N), the generalization error after T weight update steps, (TT) is:
Vn 2
g(T)= == EnND [(y Fo(x)= 15)
S 0 0 (ra(E(ff))
where rg(E(fo)) = i A
2i=d Li
(2)
2i=1
TM
A
Di=1
Here, m = min(n, D) is the rank of E,, (fo).
Notably, r (E(6)) can be thought of as a measure of effective rank of the covariance matrix and grows with decreasing values of a. Therefore, the upper bound for generalization error is higher for lower a. Taken together, Theorems 2.1 and 2.2 suggest that a can neither be too high nor too low and there exists a tradeoff region which results in efficient learning and strong generalizability. where these representations form a good basis for gradient- based optimization on downstream task performance.

---

Train and Test variation with draam (dim 1024, #train samples 200)
10"
0,0180
0.0175
Test MSE Loss
0.0170
MSE
10
0.0185
10%
10"
0.0155
2.0
Rrai
Rovan
Figure 2: Overparameterized linear regression, with inputs drawn from Gaussian distribution with power-law the covariance matrix. Models are trained with gradient descent with initialization 40 Note that particularly suffers from high train, test MSE loss, with low generalization error with a x 1.
Another motivating example Before exploring the link between a and generalization performance of deep networks, we validate our theorems in simple linear regression setting.
We first consider its relationship the finite dimensional regression setting as [11]. Specifically, Theorem 6 of their paper states that the conditions for "benign overfitting". wherein a model can perfectly fit noisy training data without any subsequent loss of performance on testing data, may be looser in finite dimensions as opposed to the necessary and sufficient condition of a = 1 in infinite dimensions. To empirically test this in the finite high dimensional setting, we examine linear least squares regression using different covariate structures for the input data. Formally, we consider covariates (a; ) such that x2 I&d is sampled from Gaussian distribution with covariance structure Σ (iiag(A,....))
where X, PL(a), i.e Xÿ & cj-a. We assume access to the corresponding labels { generated under teacher function 6", such that = x 0 + We find that in this scenario there clear relationship between the proximity of a to I and the presence of benign overfitting. As shown in Fig. 2, when is close to 1, the training loss is low, but the validation loss is also low. Thus, when a is close 1, the generalization properties are their best. This example thus provides another hint that a is potential measure for how well a model will be able to generalize.
Experimental Setup
Our theoretical results suggest the following: for representations with power-law characteristics, the decay-coefficier (a) effects the adaptivity, where the convergence rate of linear regression with gradient descent scales exponentially Onn)) with a. On the other hand, if trained sufficiently long, the upper bound for generalization error improves with larger a. Together, these items imply that if one wants to use gradient descent to train downstream task then for the representations used
A
B
supervised 0.788)'
resnet18
vgg13 vag
vit.
barlownwins 0.938)
resnet50
vit. large
100
simdrr =p/00:9407
110
resmet101
vgg
vit huge
accuracy
byol (p = 0.974y
80
50
0.6
0.8 a (decay coefficient)
1.0
1.2
0.6
14 α (decay coefficient)
1.0
1.8
Figure 3: is predictive of out-of-distribune object recognition performance. is strongly correlated to object recognition performance on STL10 across different architectures and pretraining loss functions.
5

---

A
B
supervised (p- 0.28
reenett8 8
Vgg13
vRL LLbeee
barlowwinss (p-
resswe50
1op16
viLlarge
60
simelr [p 0,803y
recdatt
Vaa19
MRt Hugge
byo (p 0.654)"
80
accuracy
60
40
40
20
20
0.6
0.8
1.0
1.2
1.0
1.4 (decay coefficient)
1.8
a (decay coefficient)
Figure 4: a is predictive of out-of-distribution object recognition performance. is strongly correlated to object recognition performance on MIT67 across different architectures and pretraining loss functions.
should be neither too large "Goldilocks" zone. Building on these insights, we now design experiments to empirically study the potential for a to be used as a representation quality metric. We do so by measuring the relationship between and the standard metric of representation quality, i.c. downstream task generalization performance. In this work, we restrict our scope to vision tasks, specifically object classification and scene recognition. In particular, our experiments:
small, i.e. high quality representations those with in
measure the eigenspectrum decay & generalization We systematically evaluate in- distribution and out-of-distributie generalization, and simultaneously, we also measure a, across diverse choices for network architectures, pretraining learning objectives, and across multiple datasets. We then look at whether a values near some neighbourhood around I are indeed predictive of representations that generalize well to downstream tasks.
explore as a metric for model-selectin in SSL: Measuring a is label-free and computa- tionally inexpensive, as it requires single forward-pass on the downstream dataset. We thus ask, is a predictive of generalization capabilities for current SSL algorithms, and if so, it be used for model selection? We run extensive ablations on design of the BarlowTwins [1] algorithm in effort to answer this question.
3.1 Measuring eigenspectrum Decay & Generalization
Evaluation Protocol: To measure for the learned representation f(x:). first extract features from intermediate layers of a DNN pretrained on Ppretrain and estimate the corresponding covariance matrix (f). Next, compute the full set of numerical eigenvalues, and estimate by a fitting linear model on the eigenspectrum in log-log scale. Our pretrained models are taken from Py PyTorch Hub [14] and timm [15]. Given that observed significant difference between the observed values in the train and test sets, we refer to this empirical estimate as the a for the dataset.
To estimate the capacity of intermediate representations in solving the downstream task, we train a linear readout layer, g(.), from representations to target logits. Intuitively, this comes down to establishing relationship between the manifold geometry and linear separability of representations [16]. Thereafter, we observe the correlation between estimated a and the linear readout performance. Notably, we also tried non-linear g()) and observed similar trend in results (see Appendix B.
Inductive bias of Model Architecture this section, investigate the relationship between a of the representation covariance matrix and object/scene recognition performance in DNNs with different backbones and the role of depth. In order to do so, we examine varying depth configurations within network architectures across three generations of models on the STL10 and MIT67 dataset [17]. We choose both object and scene recognition task as they fundamentally differ in the nature of the features that must be extracted to perform well. For scene recognition, the model focuses on global features the entire image, while for object recognition, the model focuses on local features of the image containing the object[18].

---

We examine deep Convolutional Neural Networks (CNNs) without any residual connection as our first family of models. Specifically, we choose three different configurations of VGG-Net [19], namely VGG-13, VGG-16 and VGG-19. We inspect representations that are input to the dropout and MaxPool layers during the forward pass of the network. Second, we consider Deep Residual Networks [20] which are widely used in computer vision. We inspect the representations that are input to each of the residual blocks as well as the Adaptive average pool in ResNet- 13, ResNet-50 and ResNet-100 101 during their respective forward passes. Finally, owing to the recent success of transformers in object recognition tasks we consider Vision Transformers (ViT) [21] as the third family of models, namely ViT-Base/8, ViT-Large/16 and ViT- VTT-H4ge/14. Unlike VGG and ResNet, only look at features in the intermediate layers because summarizes the entire input image and is used in practice for class prediction. For all model architectures, use the weights obtained from pretraining on ImageNet.
Fig. 3 and Fig. 4 illustrate the relation between performance and a for all the nine architectures across intermediate layer representations, as described above. We found that, while most intermediate representations in CNNs (with or without residual connections) exhibit a < 1, representations in ViTs mostly exhibit a > 1. Nevertheless, representations extracted from the deepest layers of all the models exhibit a close to 1, irrespective of the total depth of each model (see Appendix B). Furthermore, the performance on downstream task increases with depth. This is unsurprising because all networks were trained to perform object recognition hierarchical processing to learn features that are tuned towards object recognition. Surprisingly enough, we observe strong significant correlation between and performance i.e. a different data distribution than the training dataset, across layers and model architectures (p 009222, "p < 0.05 for representations exhibiting > 1 and p -0.922, "p 0.05 for representations exhibiting a < 1). It is worth noting here that the correlation was weaker for the earliest layers of each model. We believe that early layers learn more task invariant features that reflect the statistics of natural images [23, 24] and therefore lack task relevant information in their representations. Taken together, this observation confirms our hypothesis that a is a good indicator of out-of-distribun generalization performance when representations possess task relevant information. Thus, a has a necessary but not sufficient relationship with generalization (Appendix C).
ImageNet [22] and thereby would have leveraged
the STL10 dataset,
Learning objective In this section, we first aim to understand how the a value changes across the layers of fixed architecture DNN when trained with different learning objectives. We take ResNet-50 model [20] pre-trained using three different SSL. algorithms, namely SimCLR (2), BYOL [25] and Barlow Twins [1], and the supervised learning loss objectives ImageNet-1k([2]) dataset. We use a similar procedure as before to extract representations from the network and estimate a.
Similar to results in the previous section, all networks irrespective of the pretraining loss function, exhibit a closer to I in the deeper layers in contrast to intermediate layers (see Appendix Fig. 9). This surprising result indicates that although the pre-training loss function was different, representations extracted from deepest layers are reflective of the object/scene semantics in natural images. Further- more, Fig. 3 and Fig. 4 illustrate the strong correlation between a and generalization performance on STL10 and MIT67 across all pre- pre-tinin loss functions. Together with results from the previous section, we validate hypothesis that the representations that demonstrate good dittrr-dittiiutioon generalization performance are characterized by a in the neighbourhood of 1. This suggests that high quality representations are indeed those with a in this region.
3.2
-abel-agnostic metric for Model selection
With empirical evidence of a being good measure for generalization, we now wish to study its suitability as a metric for identifying the best among models pretrained with SSL algorithms. A label- free metric like could be beneficial when we don't 't have access to the downstream task annotations, and the SSL loss is not useful to distinguish models with good generalization performance (see Fig. 5). To investigate this, we exhaustively ablate the relationship between and model performance across a wide range of eeper-parameters for a representative non-contrastive SSL algorithm.
Current SSL algorithms struggle with dimension collapse (characterized by large a), due to patholo- gies in training dynamics. To study a across a wide-range of values, for our model selection experiments, we pick the Barlow Tw non-contrastive SSL algorithm that explicitly optimizes to avoid dimension collapse. particular, the Barlow Twins learning objective (LBT) proposes imposing a
constraint
7

---

A
pdlm--normalized
Set
B
test accuracy
C 4098
Ccioecy cosfficient)
4098-
4036
projector dim
2048
2045
812
512
512
0.001
0.01
0.1
0.5
0.001
0.01
6.1
0.001
0.11
0.1
0.5
(redundancy confficient)
A (redundancy confficient)
A (redundancy coofficient)
CIFAR10
pdim-normalized Pet
test accuracy
Of(eeayy coefficient)
projector dim
4096
4096
4096
2048
612
512
512
128
128
120
10"
0.01
10
0.01
0.5
10
0.91
0.5
(redundancy confficiett
À (redundancy confficient)
(redundancy confficient)
STL10
2
70
75
80
85
90
0.8
1.3
1.7
2.2
(Ver)/proj_dim
accuracy (%)
Ol(docay coefficient)
Figure 5: a as metric to inform model selection. The SSL loss (training for same number of gradient steps) is no longer useful to distinguish models with superior downstream performance. However, decay coefficient shows strong correspondence to downstream test accuracy over large hyperparameter ranges, Measuring in-distribution generalization for (A-C) BarlowTwing trained and evaluated on CIFARIO. (D-F) BarlowTwins trained and evaluated on STL10.
s.t. C(fo)is = Σn () (o
CBT Σ - C(E))))) +
C( JA
(3)
VIn fo(x En fo (x
invariance
rcdundany-rrcuuciion
Notably with sufficient large A, the model would impose an a = 0 constraint on the representations [26]. Despite the intuitive connection between and X, it is unclear whether this relationship holds across a wide-range of values for model hyperparameters. To empirically establish this connection, vary (redundancy coeffiecient), projection head dimensionality and learning rate, and train ResNet50 encoder using Barlow Twins learning objective. We provide the results for CIFAR10 [27] and STL10 [28] in Fig. 5 demonstrating that is strong indicator of in-distribution generalization performance across large range of hyperparameter ranges of LTT. We defer the reader to Appendix Fig. 14 & Fig. 15 for similar trends in optimization hyperparameters. Furthermore, of out-of-distribution generalization performance in these settings (see Appendix B.3). The same relation also holds in contrastive SSL frameworks, specifically for SimCLR [2] (see Appendix B.4).
predictive
Model Selection on Compute Bud- get: With constrained compute bud- get, using a for model selection is significantly cheaper than evaluating the representations on suite of down- stream tasks. To illustrate this, con- sider the standard alternative of train- ing a linear classifier on the represen- tations and evaluating test accuracy. Compared training linear probe, which requires multiple epochs of for- ward & backward passes through the training dataset's 's features, to achieve reasonable estimates of downstream accuracy, computing a requires a single PCA step on valida- tion dataset's features. In Table 1, we contrast the compute times for and evaluating linear probe. detailed algorithm to use a for model selection provided Algorithm and its complexity analysis in Appendix B.6.
Table 1: We report the compute time for CIFAR10, STI.10, and Imtge While CIFAR 10 and STL10 are trained for 200 epochs for downstream classification ImageNet is trained for 100 epochs. (tested in A100)
w/ eigenspectrum coefficient (a)
w/ linear probe
perf. gains
Dataset
CIFARIO (s)
o=
48=
16x
STL10 (s)
5_
80_ 10
16x
ImageNet (mins)
0.8
58 5
15x
8

---

Algorithm Model selection using a
# M: Number of models that can be trained in parallel Number sequential steps model training
# K: Number of top models to log in menory for model selection
Omin -
Omar oc models_dict {
iter range (H):
# train N models parallel, each with different hyperparam config trained_models train_8SL. (ww.aaadded......a.a.........ee
# evaluate a for the trained models
alpha_ traaaattaieedd_ddee
evaluate evaluate_alpha (Ippaaattatraieddnoottl
best wetttraind_moeet = {model: alpha for model, alpha in alpha
traine_ddain
[amin [amin,amaa amaz]]
models_stat {model: run_linear_evv (model) for model, alphe in best__rriied__odell ())
# Update amin and Omas
Omin (10m Om ay accm acej m.j. modele_stat}) Omer = min ((0m
< <a & accm acy V m. E models_stat))
# Trim models_atat to keep only top K models threshold get_best._mdell (model, mmodel_statiste statistic, topk-K) models_stat {model: threshold)
for model,
in models_stat, items if
# Return best model performance from the 'selected' models best_model_ase max ((acem m modele_stat)) return best_model_ace
4 Related Work
Evaluating representations and model quality We note a substantial body of work aiming to empirically characterize the structure of emergent representations in DNN without requiring labels [29, 30]. One such index that quantifies the similarity of representations across layers (of the same or different models) is Centered Kernel Alignment (CKA) [23]. While CKA does not provide explicit guidance for downstream performance, [31 shows that the in-distribution generalization gap can be predicted using different index based on the model'ss parameters. In particular, they show that the Empirical Spectral Density (ESD) of weight matrices for many DNNs obey a power-law, with the decay coefficient being predictive of in-distribution performance. In the present work, we explore similar indices that potentially correlate with out-of-distribution generalization by examining the eigenspectrum of activations.
Generalization in Overparameterized Models Modern neural networks often have significantly more parameters than the number of training samples, challenging the classical understanding of the bias-variance tradeoff. Overparameterization permits neural networks to overfit to noise in training data without impairing their generalization to unseen data. In recent work, Bubeck et al. proved that overparameterizztion a necessary condition for smooth interpolation in neural networks [32].
Furthermore, this benign overfitting phenomenon in an overparameterized linear regression problem been linked to the power law coefficient of the input covariance matrix [11]. Specifically, Bartlett el al. showed that benign overfitting is possible for an infiniteddimensional linear regression problem iff the eigenspectrum satisfies a power law (up to polylog factors). More recently. Lee et al. found that the tail eigenvalues of infinite- width network kernels exhibit power law decay [33]. Following this, Tripuraneni et al. explored high- high-dimensional random feature regression settings and analytically showed a dependence between the eigenspectrum decay rate of the feature covariance matrix and generalization error [34]. While these characterizations provide a theoretical understanding of generalization error in the asymptotic random feature settings, corresponding questions in the liite-dimenssonal DNN trained with gradient descent are open problems.
For deep linear networks trained using gradient descent, the eigenvalues of input covariance determine the generalization error dynamics [12]. Advani al. demonstrated that small eigenvalues determine the convergence of training dynamics well the overfitting error at convergence. For overpa-

---

rameterized 2-layer neural networks, Arora et al. provided fine-grained analysis of generalization bounds [35]. In contrast, we study modern DNN architectures and explore the covariance structure of their learned features on visual recognition tasks.
5 Discussion
Summary Our experiments suggest strong correlation between the decay coefficient for sample eigenspectrum of representations, a, and both in-distribution and out-of-distribution generalization performance on tasks central computer vision.
Representation Quality in High-Dimensional SSL We demonstrate that assessing the quality of a pretrained model with our label-free metric (a) is consistent with downstream generalization. While being computationally efficient, our procedure removes the dependence on labels, making the model selection more robust and amenable for privacy critical applications.
Necessary, but not sufficient condition Notably, a task-agnostic measure like a is a necessary but not sufficient condition for assessing good performance. To elaborate this point, let present a thought experiment. Suppose we have our ideal representation space that satisfies our claims of exhibiting alpha close to the Goldilocks zone. If we perform set of permutation operations on individual datapoint representations, the structure of the representation space remains the same, but the mapping from representation to label is now destroyed. In doing this set of permutations, have now arrived at a different set of representations that would exhibit the same (or similar) alpha but demonstrate lower task performance when measured using a linear readout layer. A similar argument is presented as theorem C. in [36]. Extending the conclusions of this thought experiment to our observations, it is clear that could have models with similar alpha values but distinctly different performances. But an alpha value that is not in the Goldilocks zone would be associated with inferior model performance. This property is the core of our claim and allows us to propose alpha as measure for model selection in SSL pipelines.
Redundancy in VIT representations Unlike other models that we considered, representations from early layers of ViT had a rapid eigenspectrum decay with a >> (see Fig. and Fig. 4). The transformer architecture has notable difference by design, i.c. early layers possess global receptive field context via self-attention on patch embeddings. Raghu et al. found that early ViT layers incorporate both local and global information [30]. Based these insights, one intuitive interpretation of our results is that the representations have a low effective rank and encode redundant information relevant across multiple scales.
Limitations While the role of and its relationship to generalization performance is better understood in the asymptotic setting for linear regression, similar questions in finiteddmmensionl nonlinear models are unanswered. It is also worth noting that the empirical correlation was weaker for the carliest layers in each model. We believe that early layers learn more task invariant features such as corners and edges [23, 24], and thereby lack rich semantic information for the fine-grained downstream task. In this case, distinguishing between poorly-trained models may be inconsistent with a.
Future Directions Learning efficiently at scale from unlabelled datasets poses an exciting open problem in deep representation learning. We hope this work encourages new perspectives into model selection for SSL pipelines and informs the design of learning objectives and model architectures to learn lusk-agnostic, adaptive features. Understanding the behaviour of (notably, scale- invariant metric) high-dimensional representation learning might provide insight into developing a theoretically grounded understanding of generalization in deep neural networks.
Acknowledgements
The authors would like to thank Zahraa Chorghay and Colleen Gillon for their aesthetic contribution
to the manuscript and figures. This research was enabled in part by support provided by Mila and Compute Canada. This work was supported by Vanier Canada Graduate scholarship (AG); Healthy Brains, Healthy Lives (AG & BAR); NSERC, Grant No. RGPIN-2020-05 and RGPAS- 2020-00031 (BAR); and CIFAR, Canada AI Chair & Learning in Machines and Brains Fellowship (BAR).
10

---

References
[1] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In International Conference Machine Learn- ing, pages 12310-12320. PMLR, 2021.
[2] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[3] Yuandong Tian, Lantao Yu, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning with dual deep networks. arXiv preprint arXiv:2010.00578, 2020.
[4] Daniel Felleman and David C. Van Essen. Distributed hierarchical processing in the primate cerebral cortex. Cerebral cortex (New York, NY: 199/), 1(1):1-47, 1991.
[5] Claus C Hilgetag and Alexandros Goulas. 'hierarchy' in the organization of brain networks. Philosophical Transactions of the Royal Society B, 375(1796):2019331, 2020.
[6] Carsen Stringer, Marius Pachitariu, Nicholas Steinmetz, Matteo Carandini, and Kenneth D Har- ris. High-dimensional geometry of population responses in visual cortex. Nature, 571(6765):361- 365, 2019.
[7] Nathan CL Kong, Eshed Margalit, Justin I. Gardner, and Anthony Norcia. Increasing neural network robustness improves match to macaque v1 cigenspectrum, spatial frequency preference and predictivity. PLOS Computational Biology, 18(1)=c1009739, 2022.
[8] J Nassar, P Sokol, S Chang, and K Harris. On 1/n neural representation and robustness. Advances Neural Information Processing Systems (NeurIPS), 2020.
[9] Zijian Jiang, Jianwen Zhou, and Haiping Huang. Relationship between manifold smoothness and adversarial vulnerability in deep learning with local errors. Chinese Physies B, 30(4):048702. 2021.
[10] Zeke Xie, Qian- Yuan Tang, Yunfeng Cai,, Mingming Sun, and Ping Li. On the power-law spectrum in deep learning: A bridge to protein science. arXiv preprint arXiv:2201. /300/1,,1111,,, 2022.
[11] Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117643):3003--3077 2020.
[12] Madhu Advani, Andrew Saxe, and Haim Sompolinsky. High-dimensional dynamics of generalization error in neural networks. Neural Networks, 132:428 2020.
[13] Vatsal Shah, Anastasios Kyrillidis, and Sujay Sanghavi. Minimum norm solutions do not always generalize well for over- wwer-parameterizae problems. stat, 1050:16, 2018.
[14] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjic Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8085 Curran Associates, Inc., 2019.
[15] Ross Wightman. Pytorch image models. 2019.
https://github. wwt.wwcww/ww.eeeet...eto.ooe...e con/rwwghttan//
serrrch-image-mone
[16] SueYeon Chung, Daniel D Lee, and Haim Sompolinsky. Classification and geometry of general perceptual manifolds. Physical Review X, 8(3):031003, 2018.
[17] Ariadna Quattoni and Antonio Torralba. Recognizing indoor scenes. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 413-420, 2009.

---

[18] A Oliva and A Torralba. Chapter 2 building the gist of a scene: the role of global image features recognition. Progress in Brain Research, pages 23-36, 2006.
[19] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprini 14KKv::449.1550 2014.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[21] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 1 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
[22] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pages 248-255, 2009.
[23] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. International Conference on Machine Learning, pages 3519-3529. PMLR, 2019.
[24] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 318-833. Springer. 2014.
[25] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent-a new approach to self -ulf-super learning. Advances in Neural Information Processing Systems, 33:21271 1112271-12244, 2020.
[26] Bobby He and Mete Ozay. Exploring the gap between collapsed & whitened features in self- supervised learning. In International Conference on Machine Learning, pages 8613-8634. PMLR, 2022.
[27] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Master's thesis, 2009.
[28] Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsuper- vised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215 223.. JMLR Workshop and Conference Proceedings, 2011.
[29] Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. arXiv preprint arXiv:2010. 15327, 2020.
[30] Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in Neural Information Processing Systems, 34, 2021.
[31] Charles H Martin, Tongsu Serena Peng, and Michael W Mahoney. Predicting trends in the quality of state-of-the-an neural networks without access to training or testing data. Nature Communications, 12(1): 1-13, 2021.
[32] Sébastien Bubeck and Mark Sellke. A universal law of robustness via isoperimetry. arXiv preprint arXiv:2 /2806, 2021.
[33] Jaehoon Lee, Samuel Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein, Finite versus infinite neural networks: an empirical study. Advances in Neural Information Processing Systems, 13:15166-15122 2020.
[34] Nilesh Tripuraneni, Ben Adlam, and Jeffrey Pennington. Covariate shift high-dimensional random feature regression. arXiv preprint arXiv:2111.0823: 2021.
12

---

[35] Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of op- timization and generalization for overparamettrizeed two-layer neural networks. International Conference on Machine Learning, pages 322-332. PMLR, 2019.
[36]
Nikunj Saunshi, Jordan Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham Kakade, and Akshay Krishnamurthy. Understanding contrastive learning requires incorporating inductive biases. In Kamalika Chaudhuri. Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 19250-19288 PMLR, 17-23 Jul 2022.
13


Reviews:
[{"review_id": "moA1_6Uoyo8", "reviewer": "Reviewer_eRnP", "summary": "The paper aims at finding a way to measure the quality of the representations learned in self-supervised learning (SSL), without having to evaluate them on downstream tasks. Based on recent experiments concerning the variance of neural representations in mammal brains, they consider the decay of the eigenspectrum of the representations covariance to evaluate those representations.  Through theoretical results, they suggest that the decay should be neither too quick nor too slow to enable downstream predictors that can be trained quickly but generalize well. Experimentally, they seem to suggest that the best representations always have a similar decay and this decay can thus be used as a metric for SSL. Furthermore, this decay is close to the one previously found in mammal brains.", "questions": "-  what are the blue lines in figure 3b and 4b? are they linear regressions on every point before and after $\\alpha=1$? in particular does the left ones take into account the ViT models on the left of $\\alpha=1$, which show the inverse correlation?\n- how are you training the linear predictor? Are you encoding the images once before training or at every epoch?\n\n**Minor suggestions/mistakes**(no need to answer)\n- please use \\citet when the reference is part of the sentence\n- line 46: the citation for [7] should be [8] I believe, ie, you cite the wrong article. \n- line 75: agnostic and metric -> agnostic metric\n- line 84: With -> with\n- equation 1: should be $f\\_{\\theta^*} = \\dots$\n- line 93: is target -> is the target\n- line 96: why do you not remove the mean feature when computing the empirical covariance matrix? I.e. this is not the empirical covariance matric, right?\n- line 110: \"Regression : Recent\" -> \"Regression: recent\"\n- line 114 : upto -> up to\n- line 227: \"all the models exhibit \\alpha,$ is unclear\n- line 270: Figure Fig. -> Fig.\n- line 273: Furthermore, there is $\\alpha$ -> Furthermore, \\alpha\n- figure 5: why are there missing values (in white)?\n\n\n", "limitations": "The main limitations of the proposed method are that it seems to be: (1) highly dependent on the architecture; (2) slow despite that being the original motivation. \n\nThe former is somewhat discussed in the paper. The latter is not sufficiently well discussed assuming that linear probing is much quicker than suggested in Table 1 (see weaknesses above).", "rating": 6, "confidence": 3, "soundness": 1, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Update**: As pointed out by all reviewers (me included) I think that there are valid concerns about using the metric proposed by the authors for selecting SSL representations. For me, the main contribution of the paper is not that much about having some new metric for model selection, but rather providing experimental and theoretical insights into understanding what constitutes useful SSL representations. \n\n---\n\n**Strengths**:\n- **Well written**: the paper is clear and was a pleasure to read.\n- **Problem well-motivated and of interest** the goal of evaluating representations efficiently is of interest to the ML community.\n- **These insights could help to understand and design SSL**: even if the method is never applied as a metric for SSL, I think that it sheds light on important properties of desirable representations that could be of interest for designing or theoretically understanding SSL algorithms.\n- **Connects ML and neuroscience** this paper sheds light on the recent findings in neuroscience that could be of interest to the representation learning community. These findings are the starting point of the paper (not a contribution) but disseminating these interesting findings to the ML community has its own value. \n\n**weaknesses**\n- **Doubts about computational gains** the main selling point of the method is its computational efficiency, yet it is not clear that there are major advantages compared to linear probing. In particular, the large computing time for linear probing (26 hours for ImageNet) makes me doubt those results. My guess is that you are not prefeaturizing the representations, ie, encoding each image only once before training and then saving the features. In my experience prefeaturization + training a linear predictor takes much less than an hour on a single GPU and 16-bit precision. Please correct me if I'm wrong, but if this is the case, then the advantage of using $\\alpha$ as a metric is much less clear.\n- **Only useful for a given architecture** In figures 3 and 4 (a and b) performance of representations extracted by CNNs seem to always be strongly positively correlated with $\\alpha$ while $ViT$ are inversely correlated. If one was to use $alpha\\approx 1$ for choosing representations in ViTs you would be quite wrong (around 10 accuracy points off). This suggests that one can only use $alpha$ to compare representations learned using similar encoders.\n- **Some experimental evidence is sometimes misleading**  I find figures 3 and 4 to be somewhat misleading, for example for each architecture there is no sign of an inflection point around and so one can change the value of $\\alpha$ by changing the ratio of CNNs vs ViTs. Furthermore, the plots and correlations are somewhat arbitrary, eg, in figure 4b if you plotted and computed the same correlations for $alpha =0.8$ or $0.9$ you would get high correlations (maybe even higher than currently). Table 1 is also misleading if no prefeaturization is applied.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PHNvNklYiEu", "reviewer": "Reviewer_uCfg", "summary": "The authors propose a fairly simple task-agnostic way of assessing the quality of NN representations. The idea is to evaluate the power-law exponent (alpha) of the spectrum of the empirical representation covariances produced by the network. Connecting with some recent theoretical results (that were however derived in limited settings), the authors argue that alpha close to 1 is correlated with good convergence and generalization properties. For this reason, the authors propose the extrapolation of this exponent as a possible metric of NN performance, of representation quality, and for model selection.\n\nAFTER REBUTTAL\nThe authors have been very open to discussing their results and addressed most of the concerns raised by the reviewers. While in my opinion some further work is needed in order to prove that the proposed measure can truly be impactful, the present study explores an unconventional and interesting direction for developing new metrics for evaluating representations.", "questions": "- Shouldn't the means be subtracted when the covariances are evaluated (eq. unnumbered after (1))?\n- Theorem 2.2: given that the generalization bound is not tight, is this result truly informative of the best value for alpha?\n- The results shown in figure 3 seem to indicate the claimed importance of alpha=1 only qualitatively, but the behavior is not at all clean. How can alpha be used for model selection or performance assessment if the power-law behavior is only an approximation, and if the extrapolated exponents can hover around 1 with alternating perfromances?\n- Plots in figures 3-4 are very crowded and the descriptions are too far from their appearance. \n- line 232-233: there seems to be a typo in the correlations, getting the same value on the two sides seems very unlikely.\n- In figure 5, the \"connection\" between alpha and the test error is very hard to assess in this way. The plots are coarse-grained and the color map is unhelpful (wouldn't be better, for alpha, to assign a specific color to 1, so that the \"ideal\" region is highlighted?). Is the relationship still just qualitative, or is it exactly true that the closer one gets to 1 the better it is? If not, how can this observation become useful in practice?\n- Did the authors try to explicitly enforce the power-law behavior with the desired exponent at training, even on a small model? Is it possible to actually test the thesis presented in this paper in a more scientific way?", "limitations": "The limitations of this work seem to reside in its very qualitative nature. In my opinion, the authors do not address nor acknowledge this issue, and the work would need a major revision in order to accommodate for it. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The idea of looking at the spectra of empirical representation covariances seems in line with a series of recent theoretical works and seems to explore an interesting research direction. \nHowever, what the authors propose in practice is based on a very strong assumption on the eigenvalue distribution, which admittedly does not hold in general. Moreover, the presented numerical results do not show a clear quantitative behavior, aligned with the authors' speculations, but just a qualitative agreement. If this method is to be seriously employed for model selection, I believe a more scientific inquiry, where the authors try to disprove their assumptions, could really help shed some light on the very complicated phenomenon of representation learning.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OJbTpWSoBjv", "reviewer": "Reviewer_78Rr", "summary": "The paper aims to find a task-agnostic statistical measure for SSL models that can serve as a good indicator of downstream performance.\nWith the observation that the eigenvalues of the covariance matrix of the features follow a power law, this work proposes to use the decay rate of the eigenvalue spectrum $\\alpha$ (i.e. $\\lambda_i \\propto i^{-\\alpha}$).\n\nTheoretical analyses (Sec 2) suggest that there should be a sweetspot for $\\alpha$, from trading off the convergence time and the generalization error.\n\nEmpirical study (Sec 3) are performed on STL-10 and MIT-67 datasets, with VGG/ResNet/ViT model pretrained using Barlow Twins/SimCLR/BYOL objective. The results show that there's a clear correlation between $\\alpha$ and the task performance (in-distribution, or transfer).\n\n=== **Update post rebuttal** ===\n\nMy TL;DR from the paper is that good performing models necessarily have a decay coefficient $\\alpha$ that lies within a certain range, which means $\\alpha$ can serve as a metric for model selection. Previously I'm concerned about the practical applicability of this metric, since empirical findings show that the dependency on $\\alpha$ is unclear in practice. During rebuttal, the authors have addressed most of my concerns, in particular:\n- How to find the \"inflection point\" of $\\alpha$ for different architectures & tasks: see discussion at the end of Sec 3.2 (and Appendix B.4); the idea is that $\\alpha$ can be used together with linear probe to reduce the computation budge (rather than replacing linear probe altogether, which would be impossible given that  is not a sufficient condition).\n- the strength of correlation: the author addressed this here: https://openreview.net/forum?id=ii9X4vtZGTZ&noteId=XTLkW4joEFR\n\nOverall, even though I'm still unsure how useful $\\alpha$ can help on model selection (which is the main goal of the study on assessing representation quality), I think the paper provides a relatively thorough study on one plausible solution for assessing representation quality.", "questions": "Please note that I'm using the line numbers in the full version, i.e. paper with the appendix.\n\n- Thm 2.2: $m$ is the rank, which may or may not equal to (i.e. could be smaller than) $\\min(n, D)$?\n- What's the takeaway of the training loss plot in Fig 2?\n- Line 197: \"no significant difference between the train & test set\": for OOD, would the gap between $\\alpha$ values estimated on the source and target domain be indicative?\n- Line 225, about the values of $\\alpha$ for CNNs and ViTs: are the dimensions kept the same across these architectures?\n- Line 234 mentions that the correlation between $\\alpha$ and the downstream performance is weaker at earlier layers. I wonder what this implies for the applicability of $\\alpha$ as metric: can $\\alpha$ be used for any representations, or only the representations from deeper layers? Are their other conditions that should be satisfied?\n- If we only consider $\\alpha$ values around 1, do we still see the strong correlation between $\\alpha$ and the performance?\n  - For example on Line 245, \"$\\alpha$ closer to 1 in the deeper layers\": if we consider only $\\alpha$ values calculated from the deeper layers, does the strong correlation still hold? In other words, are far-from-1 $\\alpha$ values only from earlier layers of the network, which are rarely taken as features in practice?\n- If we know that the desired value for $\\alpha$ is 1, can we regularize the model in the pretraining phase to encourage the model to reach this value (at least for in-distribution generalization)?\n- Line 265: please note that the Barlow Twins objective is applied on the features after the projection head, whereas for downstream tasks we take the features before the projection head, so there's a discrepancy.\n- For transfer performance, are $\\alpha$ values estimated on different downstream datasets similar? If different models are preferred by different datasets, is this reflected in the corresponding $\\alpha$ values?\n\nMinor points:\n- Line 202, 227, 274: it would be better to specify the appendix section.\n- Typos\n    - Line 172: a missing closing bracket.\n    - Line 245: \"Appendix Fig. 3\" links to the Fig 3 in the main paper.\n    - Line 248: \"Fig 3 and Fig 3\".\n    - Line 270: \"Figure Fig 5\".\n", "limitations": "The paper discusses the limitations and future directions. One limitation is potentially having inconsistent results between $\\alpha$ and performance when $\\alpha$ is based on features from early layers of a network.\n\nThere is no direct societal impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- The paper studies an importance question, and provides both empirical evidence and theoretical justifications.\n- The paper is clearly written and easy to follow.\n\nWeaknesses: my main concern is that it's unclear how transferrable the conclusions from the empirical results are to various different settings. It would be interesting to see whether the same trend holds; if not, then it'd be interesting to understand the conditions for which the trend holds.\n- There are no results on large-scale datasets (current experiments are on STL-10 and MIT67).\n- Currently strong correlations (Fig 3, Fig 5) are derived from all models, including those with poor accuracies. It's unclear whether the strong correlation still exist if we consider only relatively good performing models.\n- The results in Fig 4 do not show a strong correlation; there's a weaker correlation between the model performance and the $\\alpha$ value calculated from lower-layer features.\n\nSide note: please make the theorems rigorous (at least in the appendix where there's no space constraint). For instance, please state the convergence time in terms of the error tolerance, and please avoid using $\\approx$ (e.g. on line 532)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept