Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The submitted paper proposes GULP, a family of dissimilarity measures between representations. GULP is based on the squared error between predictions of a ridge regression solution on the two representations. Each value of the ridge regression parameter $\lambda$ corresponds to a dissimilarity measure in this family. It is shown that the dissimilarity measures in this family satisfy properties of a pseudometric. Connections and differences to CKA and CCA are discussed in detail. Experiments study if similarity of representations in terms of GULP corresponds to generalization performance of linear regression and logistic regression on these representations, if similar architectures find similar representations, and if networks with the same architecture but different initializations find similar representations after training on the same task.
strengths_and_weaknesses: The provided metric has desirable properties of a dissimilarity measure of neural networks (invariance to rotation and isotropic scaling and capturing similarities in architecture) and the paper provides an unbiased estimator. The presentation is clear and related work is extensively discussed. I have not checked the proofs in the appendix but the math in the main paper is correct. I am voting for acceptance.

The following are my major comments:

1. One of the claims is that GULP captures generalization performance of linear regression. In Section 4.1, this claim is tested only on images with random labels. I find this one experiment unconvincing and I recommend either improving this experiment or rescinding the claim. Generalization and other aspects of the model's performance can be completely different if a real-world task is used instead of zero-mean Gaussian labels. Unfortunately there are not many benchmarks for regression compared to classification, which might be why the authors chose this synthetic task. There are some well-known datasets on age estimation that can be used in this section.

2. CKA provides the stated benefits of GULP other than satisfying pseudometric properties. A discussion on why it is important for a similarity measure of neural networks representations to satisfy these properties can help with motivating GULP. Are there currently any theoretical results or algorithms in this context that require these measures to be pseudometric?

3. GULP is a family of dissimilarity measures. How similar two representations are under GULP depends on how the hyperparameter $\lambda$ is tuned. This is not a critical problem to me. Similarity of representations is a loosely defined notion and different applications would call for different measures of similarity. Even CKA comes in different forms depending on its kernel (linear, RBF, etc.). For GULP, it is possible that a scalar value can be selected or constructed from this family on the specific application. I would suggest including more heuristic or theoretically grounded guidelines for choosing $\lambda$ in specific applications. 

Minor comments:
- Bringing PROCRUSTES in the main text helps with understanding the results
- line 217: "with $\lambda=1$ resembles"
------------------------------------------------------
Update: The revision provides a realistic experiment about generalization with solid results and addresses my main concern. The discussion on the benefits of a pseudometric in the response is also helpful and convincing. I increased the score to 7.
questions: See weaknesses section
limitations: See weaknesses section
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes Uniform Linear Probing (GULP), a family of distance measures for the representations learned with neural networks, considering the representation performance. GULP satisfies the triangle inequality, shows a bounded sample complexity, and ignores the difference caused by orthogonal transformations. 

GULP uses the idea of linear probing. Different from the works in literature, which require hand-selected prediction tasks, GULP naturally considers the prediction of the learned network as the target and measures the difference between predictions of two learned neural networks given the same dataset.

strengths_and_weaknesses: The paper shows several **strengths**:

1. The method proposed is new and contributes to the distance measure between representations; even the representations are trained to have different numbers of dimensions. 

2. There are theoretical results provided, suggesting the method is sound. The method shows a bounded sample complexity and stays sound when representations are different because one is an orthogonal transformation of the other, which is a common case in neural network training. 

3. There is an analysis of the sensitivity of the newly introduced parameter ($\lambda$). Both empirical and theoretical evidence are provided.

4. The submission is well organized.

Though the topic studied by this paper is important and theoretical analysis has been provided to make the proposed method sound, I have the following **concerns and questions**.

1. In the introduction, the paper claims that GULP considers the representation's performance on downstream prediction tasks. However, in Definition 1, which defines the GULP distance, the equation uses two linear transformations, $\beta_{\lambda}$ and $\gamma_{\lambda}$. According to the equation above Definition 1, $\beta_{\lambda}=argmin_{\beta} \mathbb{E}\[ (\beta^T \phi(X) - Y)^2 \] + \lambda\| \beta \|^2$, $\beta$ is learned on the given dataset, which should come from the source task. Though the paper indicates the label $Y$ is assigned randomly in Section 4.1, the input space remains the same. I assumed $\gamma$ is learned the same way but could output a representation on a different dimensional space. Downstream tasks could be different from the source task, including the input distribution. Thus the performance could be very different. Is the assumption the source task and downstream tasks remain the same input space? Otherwise, it is not clear why the learned $\beta_{\lambda}$ and $\gamma_{\lambda}$ can be predictive of the performance on downstream tasks.

2. It is not clear to me how the centralization and normalization conditions are ensured during training in practice. The matrix $\beta$ and $\gamma$ are learned with an L2 constraint, which comes with a weight $\lambda$. With a more strict constraint, i.e., a larger $\lambda$, it seems like it would be harder to meet the request that the length of the learned representation is normalized to equal to 1. Assuming the ground truth $Y$ to be large and the norm of $\beta$ is restricted, the value of the learned representation will tend to be large. Or, is the condition of tuning $\lambda$ that finds the $\lambda$ that the learned representation is closest to meeting the centralization and normalization assumption?

3. There is also another concern related to the point above. Different $\lambda$ could lead to very different distance measures (Figure 2). Therefore an inappropriate $\lambda$ will affect the following evaluation of the distance between representations, thus affecting the empirical result of the proposed method.


questions: Please see **Strengths And Weaknesses**. One concern is related to the question above it, so I put them together.
limitations: Yes,  the authors addressed the limitations and potential negative societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Main Points:
- Provide a distance metric that captures the usefulness of a representation for the downstream prediction task
- The GULP metric measures the discrepancy between a pair of representations obtained via ridge regression estimators.
- GULP is invariant to orthogonal transformations, and is zero between two orthogonal representations

strengths_and_weaknesses: Strengths
- Proposing metrics for evaluating embedding is an important area for progressing AI
- Suitable theoretical analysis is provided for the properties of the metric
- Suitable related works are reviewed and compared
- Suitable experimental setups and results are provided

Weaknesses
- Various related works are reported in the introduction, however only 3 of these are compared in the experiments.

questions: Should you compare your method with probing methods in the NLP literature ? https://nlp.stanford.edu/~johnhew/interpreting-probes.html
limitations: Can’t find anything other than the ones mentioned by the authors.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces GULP (a pseudometric) for comparing the representations learned by two different neural networks. The core idea is to use a downstream task on the learned representations to derive the distance function. The performance on downstream tasks generally determines the usefulness of representation. So, I think it is a good idea to rely on a downstream task when comparing the representation space of two different networks.

Authors use ridge regression as a downstream task. Each sample is assigned a random target label drawn from the Normal distribution, and a linear ridge regression model is trained. The final distance is a difference in the regression estimate of two representations under all regression functions.

The authors present both theoretical and empirical results. The theoretical results derive the closed-form expression of distance and prove its structural properties. Authors use sample estimates for GULP and theoretically justify that the estimator is concentrated around its expectation. The empirical results show the benefits of GULP, such as it captures generalisation over linear predictors, can be used for comparing architectures, the evolution of distance over training, and the sensitivity analysis of networks.

While the work is interesting, I think the critical element missing is the justification behind the choice of distance. The distance is closely related to ridge-CCA. I find it hard to understand the introduced distance's essential theoretical benefits over other alternatives.

I am more concerned about the representation used for computing the distance. After the final layer of neural networks, representation retains class-specific information and throws away the rest. That is to say, in BCE loss, neural networks maximise MI(Z; Y), where Z is the representation and Y is the task. I do not see the usefulness in comparing two networks using a distance on their final layer representation for any different downstream task such as ridge regression. Perhaps comparing the representations learned by self-supervised/unsupervised methods will be more meaningful.
Overall I think it is a good paper with sound theoretical results. On the practical side, there is a scope to improve the paper. Can GULP be extended for different downstream tasks? Also, some justification for using ridge regression as a downstream task. I give a score of 6. If the authors respond sufficiently to my comments, I will be happy to adjust my score.

***Post Rebuttal***

I appreciate the authors for their effort in replying to my comments. I find it as a good paper that could be more attractive with the right set of empirical results. I prefer to stay with my original assessment.

strengths_and_weaknesses: Strengths:

Theoretically sound pseudometric for comparing representations of two architectures. The paper is well written, and the proofs are easy to follow.
Empirically demonstrate the use of GULP on applications such as grouping similar architectures, comparing the training dynamics, and the sensitivity analysis of architectures.

Weaknesses:
             
The analysis in section 4.2 relies on MNIST data. Given the complexity of features in MNIST digits, I wonder whether the information content of the features learned by various architectures differs significantly. A little more challenging data, such as cifar10, 100, etc., would be more realistic to consider. Authors later analyse architectures trained on the imagenet. However, I still think MNIST is not a good choice, even as a toy problem for comparing representation.

The empirical results can be better established. For instance, one can use GULP to compare representations of different layers of the same network to see how downstream performance improves with the depth or width of the network. Can potentially investigate its application to tasks such as knowledge distillation.

I don't understand the motivation and practicality of using GULP for network convergence. In section 4.3, 16 resnet18 models are trained, and their distance is reported under GULP and other distances. It is not apparent what the authors mean by distances capture fine-grained differences. The same model should learn similar features under different initialisation for the same task. One could report mean and std in the loss for this purpose. What is an added gain to looking at "fine-grained differences in features". After all, the distance comes at an additional computational cost.

Minor Comment: The choice of notation could have been better. It is easier if random variables are bold, likewise use a standard choice for scalar, vector, matrices, etc. The x-axis labels in dendrogram plots are too small, making them hard to interpret. It would be nice to plot these in higher resolution and move them to the appendix.

questions: In section 2, line 71, the representation map 'f' is centred to 0 mean. Next, the maps phi and xsi are introduced. It is unclear whether phi and xsi are two choices for 'f' also centred to 0 mean. If yes, then I think its wrong to say ‘xsi(x) = a phi(x) + b’ as this would imply E[xsi(x)] = b whereas the normalisation would mean E[xsi(x)] = 0.

Can GULP be used for comparing the representation of networks trained on different datasets? It can be an interesting empirical result.
limitations: The authors provide some discussion on the limitations of their work. The main downside, in my view, is that GULP relies on the ridge regression task. If the network is trained with cross-entropy loss, it is not immediately obvious what are the benefits of evaluating it under a ridge regression task. Furthermore, downstream applications have other choices such as classification, clustering, etc. The authors mention that in conclusion, but it appears to be the main issue as much of the downstream tasks are not regression. They also say GULP has a closed-form solution under kernel ridge regression. It would be better if the authors showed empirical results with kernels.
 
The empirical evaluations use final layer representation would be interesting to investigate applications of GULP for comparing intermediate representations as well as its suitability for comparing networks trained on different datasets.
review_title: null

Meta-review: metareview: All reviewers are in agreement that this is an interesting theoretical and empirical contribution and a useful tool in better understanding neural networks.
confidence: Certain
Decision: Accept