Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper performs a comparative study of three representation similarity metrics (CKA, Procrustes and PWCCA) evaluated on representations generated by AlexNet and MobileV2Net. This work extends Ding et al. (2021) (which studies the same metrics on ResNet with CIFAR-10 test images and language models). The main contribution is the reformulation of these metrics to enable efficient computation on larger models with larger images. Further contributions include the comparison of the evaluation of these metrics on two functionality measures: linear probing accuracies and network performance deficits.
strength_and_weaknesses: Strengths

The paper is well written - the theory and results are presented clearly. The paper presents interesting insights into the performance of various similarity metrics on representations generated by CNNs with two functionality measures.

Weaknesses

The novelty/originality is unclear.  The paper appears to extend the work of Ding et al. (2021) who performs similar comparisons of these metrics on ResNet with CIFAR-10 (omitting the analysis of these two functionality measures).  If the main contributions is the reformulation of the metrics and the inclusion of these two functionality measures - it would be useful to see how this compares to the work of Ding et al. (2021)  e.g. the computational resources used under the original formulations on ResNet or AlexNet vs the reformulation. 
clarity,_quality,_novelty_and_reproducibility: Some questions on clarity:
The CNNs are trained on ImageNet. Is ImageNet also used as a test set for evaluating the metrics?
How do these functionality measures compare to the study performed by Ding et al. (2021)?


summary_of_the_review: This work provides some interesting insights on representation similarity metrics evaluated on CNNs. The novelty/originality needs further clarification, especially with respect to Ding et al. (2021).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper conducts a series of experiments to study how well each metric captures changes to representations. Linear decoding probes and network performance are choosen to be the measurements. Authors also propose several modification to existing metrics to make it possible to evaluate high-diemntional representations with low computational cost. Experiments are conducted on AlexNet and MobileNetV2 pre-trained with ImageNet.

strength_and_weaknesses: **Strength:**

- The reformulation of CKA, Procrustes can be applied to high-dimentional representations.
- Some observations are new compared to existing studies.

**Weakness:**

- I think this paper is a simple extension of [1] which follows the same testing protocal. The observations are also empirical without deeper explanation.
- Apart from Image-net pretraining and classification, more experiments should be performed on different tasks to further verify the claims in this paper.
- The organization can be improved. For example, Page 8 has so much blank areas. 
clarity,_quality,_novelty_and_reproducibility: - Clarity, Quality： The organization can be further improved. 
- Novelty: I think the novelty of this paper is a little limited. 
- Reproducibility : The paper has provided the code for reproducibility.
summary_of_the_review: - This paper has merits on reformulating existing metrics for advancing calculating high-dimentional representations and conducting experiments to observe the correlation between metric and functionality. Based on the experiments, authors find some new observations. However, I think the experiments are not sufficient to support the claims. The overall contribution is not enough for accepting this paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper address a representation similarity metric analysis for high-dimension features by reformulating the representation simiarity metrics such as CKA, Procrustes, and CCA-based methods. Also the authors validate the changes by ablated representation by comparing linear decoding and non-linear classification performances. They present the analysis results on two CNNs such as AlexNet and MobileNetV2 traind on ImageNet. 

strength_and_weaknesses: ### Strength
- In-depth analysis on representation similarity is an important and fundmental topic on representation learning. It can contribute to model interpretability.
- Representation similarity metrics for high-dimensional features is also a significant topic.
- Experiments and analysis on ImageNet-scale data are valuable.

### Weakness
- [Major] This paper seems to be basically based on [Ding et al. 2021] and its extension to high-dimensionality by reformulating metrics and linear decoding analysis. However, the advantage of the high-dimensionality is not effectively showed. The relevant information on high-dimensionality is that the authors validate their method on AlexNet and MobileNetV2 trained with ImagNet only. It is hard to find the specific details on high-dimensionality despite the emphasis in the title.
- [Major]  I am not sure high-dimensionality is mandatory in represenation similarity metric analysis because a feature dimension of a single layer of many Transformer-based models is from 768 to 2048. To show the efficacy, experiments on higher dimensional feature similarity might be required. For example, to emphasize the effects of high-dimensionality, the aughors can concate two layers features over several tens of dimensionality. 
- [Major]  The authors presented 10 classes results in Figures 2 and 3. It is not clear how to select 10 classes from 1k classes of ImageNet. Overall, the experiemental setup is not clear for reproducibility and understanding even if they provided the source code. 
- [Major]  Following [Ding et al. 2021], experiments on multiple seeds are required. 
- [Minor] Although the authors argue they validate their method on popular CNN models, AlexNet and MobileNetV2 are not popular nowadays. Because [Ding et al. 2021] presents the results on BERT, the authors need to consider the similar experiments. The experiments on ViT are ok. 
clarity,_quality,_novelty_and_reproducibility: ### Clarity and Quality
- This paper is not easy to follow. Overall flow of this paper seems to follow [Ding et al. 2021]. But the paper is not typical organization. For example, Conclusion looks far from highlighting the core arguments of the paper. 
- There are too many contribution summary. Indeed, some of them is hard to be considered as contributions (4th and 5th)

### Novelty
- Idea on comparing linear decoding and classification accuracy is interesting. But overall flow is based on the previous work.

### Reproducibility
- The authors submitted the source code. However, it is not easy to understand how to conduct the experiments. 
summary_of_the_review: Despite the topic importance, this paper has some room to be improved (See the weakness). However, I am not an expert in this topic, so I mainy decided my score based on paper organization.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Although the reviewers noted that the paper is well written and the topic is relevant, I do not recommend accepting the paper in its current form. The main issue, raised by all reviewers, is that the work is quite similar to Ding et al. Some of the reviewers would also have liked to see experiments with higher dimensional feature similarity, and experiments on more datasets.
justification_for_why_not_higher_score: Lack of novelty
justification_for_why_not_lower_score: n/a
Decision: Reject