Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper tries to give a measurement method to evaluate the learned model, which is highly correlated with the final test accuracy.
The measurement method depends on two key factors: Intrinsic Dimension (ID) and CLuster Learnability (CL).
This paper claims that the model with higher ID and CL performs better. Using ID and CL to predict top-1 accuracy can achieve a Pearson correlation coefficient of 0.93, which is better than existing predictors (e.g., alignment and uniformity).
Inspired by the above observation, this paper proposes a modified DeepCluster algorithm to increase the final performance.
main_review: Strengths:
-  Pearson correlation coefficient between the proposed factors (ID and CL) and final accuracy is surprisingly high. Thus, they can be a very good unsupervised monitor for self-supervised training.
- The proposed algorithm inspired by CL outperforms DeepCluster.

Weakness:
- The technical part is somewhat simple and lacks insight into the proposed factors. For example, the proposed factors seem to be independent of the learning methods and learning paradigms, since they characterize the expressiveness and learnability of pre-trained representations. However, in the top left of Fig 1, DINO has both better ID and CL than resnet50, while supervised learned resnet50 should outperform DINO. As a comparison, for Align-Unif Predictor (top right of Fig 1), resnet50 has better alignment and uniformity than dino (and all self-supervised methods), which matches the fact.
- According to the paper, higher ID indicates better expressiveness. However, in the experiments (top left and bottom left of Fig 1), resnet50 has less ID than resnet34, and resnet34 has less ID than resnet18, which is an apparent contradiction. Therefore, the motivation for seeking higher ID is not well supported.
- Furthermore, Pearson correlation coefficient as the only metric is not enough to conclude that ID and CL are better than other existing predictors. This paper needs more investigation on their scope of application and limitations.

Minor comments:
- Different models have different normalization of the feature. For example, SimCLR forces $||f||=1$ and BYOL does not. Thus, using cosine distance to infer ID is not correct, since cosine distance is equivalent to Euclidean distance iff $||f||=const$.
- Fig 1 is not a vector diagram (become fuzzy when zooming in) and there are many repetitive labels. 

summary_of_the_review: This paper is novel and the results are interesting. However, the technical part is somewhat simple, and the insight behind the observations needs further exploration.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes two metrics for assessing the quality of self-supervised learning representations in terms of expressiveness and learnability. Expressiveness expects to maximize the mutual information between the representation and the original data, while learnability emphasizes the representation suppose to be simpler and more learnable. Based on these two intuitions, the author applies Intrinsic Dimension (ID) and Cluster Learnability (CL) as the measurements of expressiveness and learnability respectively to predict downstream classification performance. The authors collect 30 checkpoints of recent self-supervised methods to validate the proposed metrics on two datasets, ImageNet and STL_10, and the results show a high correlation between the proposed metrics and the downstream classification performance.
main_review: Advantage:
1.	Current assessment methods for self-supervised learning depend on the availability of labels, while the two metrics proposed in this paper escape this dependency. The proposed metrics provide a new evaluation protocol for self-supervised learning in the case where annotation is absent.
2.	The proposed two metrics are shown to be highly correlated with the performance of the self-supervised methods on the KNN classification task, which inspires further exploration of the relationship between expressiveness-learnability and representation quality.

Disadvantage:
1. The performance of linear evaluation and KNN classification tasks on the pre-trained dataset is approaching a bottleneck, and more attention has been shifted to the transferred performance on other datasets. However, this paper only shows the correlation between classification performance on the pre-trained dataset and the proposed metrics, while ignoring the transfer performance on other datasets, which limits the impact and significance of the paper. The current application scenario is not attractive for unsupervised learning.
2.  Complexity is important for the scope of application of the method. However, the complexity of the proposed metrics is not discussed in this paper.

summary_of_the_review: 1. Section 3.2 indicates that the estimator is exact for uniformly distributed data, and datasets applied in experiments are balanced. Is the proposed metrics only work for the balanced dataset?
2. Figure1 is fuzzy and the name of methods overlap badly, which affects the reading. The description to the calculation of CL is unclear and confusing. If I understand correctly, is each sample is classified according to all the samples have seen before?
3. The improvement brought by KNNDeepCluster is tiny compared to the base method, which is hard to be convinced of the effectiveness of the proposed method.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes Intrinsic Dimension (ID) and Cluster Learnability (CL) as an alternative solution to efficiently evaluate learned representation quality of pre-trained networks, without using down-stream tasks/labels. These two frameworks are inspired by finding “expressiveness” – finding the smallest number of variables to approximate the representation; and “learnability” – number of data samples needed for KNN clustering.

Extensive experiments have been evaluated on models trained with both supervised and unsupervised methods, and results showed that ID + CL would produce a high correlation with prediction accuracy from these methods, comparing to baselines. Finally, the authors argue that CL can also be incorporated into DeepCluster (an unsupervised method), as an auxiliary loss to further improve the prediction performance.

main_review: General comments: 
The general direction and the problem setting which this paper aims to investigate are interesting, but I found that both ID and CL frameworks are heavily identical to the previous frameworks and offer no additional insights on understanding representation quality or other aspects of network learning.

Detailed comments:
-- I found that the writing and notations of this paper are quite hard to follow. And Some claims and analygies are over-exaggerated. For example, i) I don’t understand how language compositionality has anything related to the motivation of this work. ii) In contribution (section 1), the authors said that they analyze self-supervised learning using the proposed methods and then in the second paragraph, they said they evaluates on both self-supervised and supervised models. iii) In Section 3.3, a lot of terms are not properly explained: I don’t understand what $KNN(z_i|z_{k<i},\tilde{y}_{k<i})$ , and “seeing the pairs”, “online learning accuracy” really mean.  iv) I don’t think Q1 and Q2 in Section 4 are different. If high-performing models are having high CL and ID scores, and automatically it indicates that they have a high correlation?

-- Low novelty. The formulation of ID is nearly identical to the original publication from Ansuini, et al, including the explanations and notations used in Section 3.2. The formulation of KNN seems to be a straightforward extension.

-- Comparing to Align-Unif. The results in the original paper indicate that lower alignment and lower uniform would have a better prediction performance. But in the Fig 1., it seems to indicate otherwise, which is not further justified. 

-- Using CL framework to improve self-supervised methods leads to very marginal improvement on ImageNet. Without additional information such as variance, it’s hard to confirm that CL systematically improves representation learning. Further, I found this section is a bit deviate from the main focus of this work. Additional experiments, insights or visualisation on how different ID and CL could lead to different representation learning quality and limitation should be more helpful on justifying/understanding “expressiveness” and “learnability” of representation learning.

summary_of_the_review: This paper proposed Intrinsic Dimension (ID) and Cluster Learnability (CL) to estimate supervised and self-supervised model's prediction performance. However, the proposed frameworks are heavily identical to the existing works. Considering the less-structured writing and confusing notations, I believe this work has not met the publication quality.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to use Cluster Learnability (CL) and Intrinsic Dimension (ID) to evaluate the representations learned by self-supervised learning methods. The authors collected 30 checkpoints and show that their method is more predictable when compared to other methods, e.g., alignment and uniformity. Moreover, the authors modified the labels generated by K-means in DeepCluster to improve the learnability of the representations and the results demonstrate slight improvements.

main_review: strengths:

(1) This paper is generally well-written and the motivation is clear. How to better evaluate the representations learned by various self-supervised learning methods is an important question and it will give further instructions to develop new SSL algorithms. 

(2) The experimental results demonstrate the effectiveness of the proposed metric CL and ID. I appreciate the authors for their hard but valuable work to collect and evaluate 30 existing SSL checkpoints. 

----------------------------------------------------------------------------------------------------------------------------------------------------------

weakness:

(1) It is not clear why we should use CL and ID for evaluation. In other words, is it sufficient (enough) to use these two metrics? I think this work could be better to give some theoretical analysis (on the decomposition of CL and ID) instead of heuristic design. Although the experimental results show some advantages, it is still hard to understand why should we use these two metrics.

(2) The alignment and uniformity framework is more suitable for contrastive methods (by decomposing the InfoNCE loss into two terms) and it may be not an appropriate indicator for other methods, e.g., DINO. Hence, I think the comparison method is not a strong baseline. 

(3) I think it is better to design an SSL algorithm by directly optimizing CL and ID. The modifications made on DeepCluster seems a little far-fetched with proposed objective and the results are not convincing enough, especially on ImageNet. It feels to me that it is hard to formalize CL and ID as our loss objectives to optimize. Although mentioned in the conclusion, I think the authors should make more efforts along this direction to make this work more complete. 

(4) As the main result of this paper, Figure 1 is blurry. It is not clear what is the relationship between accuracy and colors of different brightness. Also, there exist some typos (e.g., 'deepcluste' should be 'deepcluster' in Figure 1). Hence, I suggest the authors to improve the quality of Figure 1.
summary_of_the_review: This paper is generally well-written and the motivation is clear. However, I still have some concerns (see weakness) and I think this paper could be more complete by providing theoretical analysis or developing an SSL algorithm which directly optimizes CL and ID.
review_title: null

Meta-review: The problem studied in this paper is interesting and the high-level motivation of the proposed research is reasonable. However, as pointed out by reviewers, it is not convincing that the developed components in the proposed method are able to address the issues mentioned in the high-level motivation. Furthermore, the experimental results are not convincing to verify the motivations either. Though the authors provided some clarifications in the rebuttal, reviewers' major concerns still remain.  

The authors are encouraged to take reviewers' concerns into consideration to revise the proposed method to make it a stronger work for future submission. Based on its current form, this work is not ready for publication at ICLR.
Decision: Reject