Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper develops and applies a set of methods to determine the functional equivalence of architecturally-identical neural networks trained with different initializations. These methods involve taking the activations from a given layer l in one network, inputting them into a "stitching layer" (either a linear projection or a 1x1 convolutional layer), then taking the activations from the stitching layer and inputting them into layer l+1 in a second network. The stitching layer is optimized either to minimize the models' loss on a task, or to minimize the distance between its own outputs and the activations of the second network. The authors find that stitching layers can typically be learned that have minimal impact on model accuracy and loss. They also demonstrate some results that they claim undermine the utility of CKA, a popular representational similarity metric.
main_review: **UPDATE: I have increased my rating from a 3 to 7 in response to the authors' responses**

I think the approach presented in this paper is clever and relevant, and could yield results that are worthy of publication. But I have a hard time feeling confident about the work in its current form. It’s difficult to assess whether the results are meaningful, both because of a lack of clear baselines and controls, and a lack of interpretation and explanation by the authors. The results are also limited to very small models and datasets. However, I am optimistic that my concerns can all be addressed through additional analyses and writing.

I applaud the authors’ use of replicates and measures of variability in plots and experiments. This is necessary but disappointingly uncommon in empirical machine learning research.

The authors open with the question “When are two representations the same?”. This statement could be refined. “The same” is a binary assessment, but the paper seems interested in measures of similarity, and similarity is a continuous quantity. I think the present work attempts to dissociate functional similarity (e.g. changes in performance measures) and representational similarity, and the authors should focus on this distinction. I don’t mean to be pedantic—I think the work is well-motivated and the problem is relevant, but the paper opens with a framing of the problem that undersells it.

One general issue I have with this paper (which recurs in some of the comments below) is a lack of clear baselines for establishing whether some change in a quantity is meaningful. For example, what is the cross-entropy between model outputs at init? Or when the stitching method is applied to two networks at init? Or when the task loss/stitching layer method is applied, but the stitching layer is trained to minimize the *change* in the model’s outputs before vs. after stitching? All of these would help contextualize the current results.  

The authors only compare their results to CKA, choosing not to use CCA-based representational similarity measures because those “indices do not succeed in identifying similar layers of identical networks trained from different initializations” [L67-68]. Just because CCA-based methods are more sensitive to differences in initialization than CCA doesn’t mean they’re irrelevant to the present work. The present work develops methods to test whether differences in initialization are meaningful, so it seems sensible to me to include other representational similarity methods as a control.

One way the authors demonstrate the potential unreliability of CKA is by plotting raw CKA scores vs. cross-entropy (Figures 4 and 5). This is difficult to interpret without some baseline or context for what magnitude change in these quantities is meaningful. It also needs to be quantified. At the very least, compute a non-parametric correlation between these quantities. This extends to other representational similarity metrics, which I would encourage using (as per the previous comment). 

The authors also demonstrate the unreliability of CKA by “stitching in” a linear layer initialized with an identity mapping, then training the layer using the task cross-entropy and a term to penalize the CKA value at the stitch. They find that they are able to maintain high accuracy while causing the CKA value to decrease. I am skeptical of this result for multiple reasons. Related to my previous comment, there is no context to determine whether this change in CKA value is meaningful (it even looks like accuracy decreases a small amount!), and the result is not quantified. Furthermore, CKA was specifically designed not to be invariant to all invertible linear transformations. Given the small changes in accuracy and cross-entropy, I suspect that the stitched-in layer simply learned a transformation to which the network is *less* sensitive than CKA. 

It’s not clear to me what the authors mean when they state that they initialize from the optimal least squares matching (e.g. Line 180). Do they mean that the activations from Network 1 are first transformed using the linear projection obtained by solving a least squares mapping between Networks 1 and 2, and then the linearly-projected activations are fed into the convolutional layer?

I think it might be helpful to attempt to dissociate the effect of the type of projection used in the stitching layer from the effect of the objective. Is it possible to solve the linear projection(s) using the task loss as the objective?

Is it possible to use the objectives of representational similarity measures (e.g. CKA, CCA) for direct matching? If so, it could demonstrate the Frankenstein framework’s utility for benchmarking representational similarity measures.

It’s not clear to me how to interpret the sparsity results. The section is titled “​​direct vs. task loss matching”, but these conditions are not directly compared in the text, and it’s not immediately apparent to me that this comparison is meaningful. Is there a meaningful baseline that could be used here? Perhaps inserting a linear layer at the same depth as the stitching layer, then training the layer to maximize both sparsity and accuracy, and examining how accuracy changes as a function of sparsity? 

Some of the results are very interesting, but nearly all of them are presented without much interpretation. For example, what is the implication that Procrustes matching yields similar results as linear least squares matching for later layers but not for early layers [L175-177]? What is the implication of the lack of mode connectivity for stitching layer solutions in early layers?

What is Layer 0 in the Resnet-20 (Figure 2) and why is performance so low when using it to stitch/match representations?

What models are used in Figures 4-8?

In Figure 8b, the x-axis is labeled “sparsity”. How is this computed?

limitations_and_societal_impact: The authors only provide three sentences on the limitations of their work. I understand space is limited, but I think they could acknowledge the limited size of the datasets they use, and some of the instances in which their results were inconsistent and/or difficult to interpret.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: **EDIT: I have increased my score from 4 to 6 based on the authors' responses and their proposed edits**

The submission considers the representational similarity between two neural networks by fitting linear transforms between corresponding layers. The authors investigate "direct mapping", where the linear transformation is learned simply based on the layer activations, and "task loss matching", where the linear transformation is fit to optimize task performance of the "Frankenstein" network (layers 1–n from model 1, layers n–end from Model 2), when holding all other network parameters fixed. They observe that a single linear 'stitching layer' can successfully map between corresponding layers of two networks of identical architecture and training, differing only in their random initialization. The performance of the Frankenstein network depends on the architecture, the fitting method (direct fit vs task loss matching), and the layer at which the stitching is performed. When tested on Inception, there is no difference between layers or fitting method. The dependence on fitting method and layer are most pronounced for ResNet-20. They compare their results to Centered Kernel Alignment to highlight that low CKA similarity can be associated with high classification accuracy. Additional experiments attempt to characterize the stitching layers.
main_review: Some related work is missing. For example, the "Frankenstein networks" described here are quite similar to the "selfer" networks in Jason Yosinki's NeurIPS2014 paper: "How transferable are features in deep neural networks?". In that work, they similarly characterize the representations in neural networks by how interchangeable the heads/tails of the networks are, although without the addition of a stitching layer. Other work has also used layer-wise linear transforms to characterize representations, for example (Alain, G., & Bengio, Y. (2016). Understanding intermediate layers using linear classifier probes. ArXiv, 1610.01644v3). Romero 2014 used a least squares loss between the activations of two networks in work on distillation (Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., and Bengio, Y. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014.) The submission does reference the Kornblith 2019 paper, but doesn't make explicit how the Dr Frankenstein framework presented here differs from the Linear Regression condition tested in that work. The authors could also consider pointing to work in neuroscience which compares layers of neural networks to regions along a sensory hierarchy in the brain via linear matching (e.g. Yamins, D. L. K., & DiCarlo, J. J. (2016). Using goal-driven deep learning models to understand sensory cortex. Nature Neuroscience, 19(3), 356–365. https://doi.org/10.1038/nn.4244).

I am unaware of another paper documenting the fact that one can learn a linear transformation between corresponding layers of two networks of identical architecture that preserves their performance. However, I disagree with the authors that this result is surprising or remarkable. They may want to elaborate on why this result is remarkable. 

The submission focuses on the strengths of their work without carefully considering its weaknesses. The comparison to CKA feels somewhat incomplete. If the proposed framework is really intended to measure representational similarity, then it should be evaluated according to the same benchmark as CKA. Does Dr. Frankenstein find corresponding layers in two networks of identical architecture and training, differing only in their random initialization, to be most similar (relative to all other pair-wise comparisons)? Given its dependence on a linear transformation between layers, I suspect it will perform similarly to linear regression, which performed significantly worse than CKA on the layer matching task in Kornblith 2019. An honest discussion of the strengths and weaknesses would discuss when one might care more about task performance and when one might care more about the similarity of representational geometry.  Since CKA is only invariant to orthogonal transformations, not any invertible linear transformation, it is not surprising that the high performance matchings found here sometimes correspond to lower CKA scores. In fact, that is by design. Representational similarity is decidedly not just about task performance. One may obtain identical task performance with completely different features. One wants a similarity metric that is sensitive to those differences. However, I permit that there are certainly cases where task performance is what matters or where some insight about the network can be gleaned from analyses similar to the one presented here. But the submission does not present this as a tradeoff; they describe this characteristic of CKA purely as a weakness.

The submission is fairly well-written. However, I find that it somewhat over-complicates a relatively simple set of ideas. 

The introduction could be improved by providing more motivation for why the authors chose to focus on matchability.

The most important contribution of this paper is the documentation of under what conditions one can arrive at a performance-preserving linear stitching layer between corresponding layers of two identical models (for the three models tested).  It is much less clear from the submission how the Dr. Frankenstein framework is useful or insightful as a method for measuring representational similarity.

limitations_and_societal_impact: The authors briefly discuss limitations but do not mention any potential negative societal impacts. No potential negative societal impacts readily come to mind.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper presents a framework to evaluate the similarity of intermediate representations from two neural networks. Given two neural networks with the same architecture but trained with different weight initializations, the framework uses a stitching layer to search for a transformation between the intermediate representations of the two networks, such that the model performance on a given task is preserved. The paper further proposed two approaches to obtain such transformations. Under this framework, the author empirically observed that representations from a given layer of two convolutional networks can be matched.
main_review: 1. A core claim made by the paper is the empirical observation that under the proposed framework, representations from a given layer of two convolutional networks with the same architecture, but different weights can be matched. This claim is supported by the results in section 5 on three convolutional architectures (Tiny-10, ResNet-20, Inception-V1) on two tasks (CIFAR-10 and CelebA). However, it is not obvious whether the matching method is generic as claimed on line 100, even to other convolutional architectures. It is also unclear how architecture, tasks, training procedure can affect the observation. More explanation and descriptions should be made to this part. 

2. I found the paper a bit hard to follow, especially the experiment section. As the proposed toolset is expected to provide an evaluation framework on representation matching, I would expect to see how the similarity metrics, namely the loss or accuracy when training with the stitching layer, is correlated with 1) matchability of the representations and 2) with the “ground truth” representation similarity. Yet the experiment section 5 is extensively about comparing different methods to obtain the stitching layer, while section 6 is only comparing with existing metrics (similarity indices).
limitations_and_societal_impact: none
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a new tool using a stitching layer to analyze the representational similarity between two deep neural networks (DNN). The idea behind the stitching layer is that it connects one frozen network to another via a parametrized layer that is optimized using the soft labels obtained from one of the frozen networks. The intuition is that if representations of two different networks are highly similar the final performance obtained using the stitching layer should be close enough to the original model's performance.   

The key experiments demonstrate the following:
1. Using the stitching layer DNNs with the same architecture but different initialization can be matched almost perfectly
2. Standard similarity measures in the literature e.g. Centered Kernel Alignment(CKA) are not indicative of final task performance using stitching layer. Even with a low value of CKA, high performance can be obtained on the target task using stitching layers.

main_review: **UPDATE: After reading the author's response and planned updates, I believe the paper will be valuable to researchers interested in applications of neural network similarity in functional usecases. Therefore I have updated my score to 7**

**Originality:** The paper introduces the idea of stitching layers to analyze similarities between neural network representations. The idea is derivative of [9] where they used a stitching layer to inspect equivalence between DNNs trained on different tasks. Here, the authors show results on comparing DNNs trained using different initializations.

The improvement on the original idea from [9] by using the optimal least-squares solution as initialization of the stitching layer is new. As shown in Figure 6, it improves the matching and therefore makes it possible to match even the deeper layers. 

The comparison with CKA showing that the similarity measure is not indicative of task performance shares new insights into the limitations of these measures.

**Quality:**
The submission seems technically sound with well-designed experiments to support the key claims. The experiments showing the importance of OLS initialization (Experiment 5) and comparison with CKA (Experiment 3 and 4) are well designed. Further, the experiments are performed on three different architectures and two tasks showing the generalizability of the claims.

However, one important piece of information that is missing is which images were used to calculate CKA. If the number of images used to calculate CKA is less the results can be noisier, so it is crucial to provide these details.

Another weakness of the paper is that it shows the cases where the stitching layers work successfully. It might also be interesting to find out where they fail, for example, if two DNNs with the same architecture are trained on different tasks how successful is transformation using the stitching layer with respect to the depth of the network. How successful will the stitching be from a randomly initialized network?


**Clarity:**
The paper is well-written with the methods clearly explained through equations and Figures. The results section is also organized well. 

**Significance:**
Identifying measures of similarity and their limitation is relevant to a wide set of applications such as transfer learning, multi-task learning, and interpreting what the network has learned. In this aspect, the paper introduces a new similarity measure based on task performance and shows the limitation of existing similarity measures.

However, the paper doesn't show where stitching layers fail to provide as good performance as the original models. Such analysis might have revealed new directions about the potential applications of the proposed measure. 
limitations_and_societal_impact: Yes
review_title: null

Meta-review: This paper examines performance when an affine layer is used to “stitch” together layers from two neural networks, so that the hidden representations of one network are used as inputs to an intermediate layer of the other. Reviewers agreed that study of performance achievable via matching with either least squares or task loss was an important contribution. However, several reviewers expressed concern regarding the framing of the work, and in particular, the lack of clarity regarding the relationship between representational similarity methods and the proposed framework. At the urging of Reviewer j6kV, the authors proposed to change the introduction to describe the relationship between representational and functional similarity and to motivate the work from this perspective. Reviewers also raised several more technical questions. The authors provided a summary of the revisions they plan to make, and after reviewing this summary, all reviewers have recommended acceptance. The AC agrees that the paper, with the proposed revisions, provides a valuable perspective on representational and functional properties of neural networks.
Decision: Accept (Poster)