Similarity and Matching of Neural Network Representations
Authors: Adrián Csiszárik
Péter Kőrösi-Szabó
Ákos K. Matszangosz
Gergely Papp
Dániel Varga
Venue: NeurIPS 2021 Poster (2021)
Keywords: Representation learning
Image representations

Abstract: We employ a toolset --- dubbed Dr. Frankenstein --- to analyse the similarity of representations in deep neural networks. With this toolset we aim to match the activations on given layers of two trained neural networks by joining them with a stitching layer. We demonstrate that the inner representations emerging in deep convolutional neural networks with the same architecture but different initialisations can be matched with a surprisingly high degree of accuracy even with a single, affine stitching layer. We choose the stitching layer from several possible classes of linear transformations and investigate their performance and properties. The task of matching representations is closely related to notions of similarity. Using this toolset we also provide a novel viewpoint on the current line of research regarding similarity indices of neural network representations: the perspective of the performance on a task.

Similarity and Matching of Neural Network Representations
Adrián Csiszárik Péter Körösi-Szabó', Ákos K. Matszangosz! Gergely Papp", Dániel Vargal
Alfréd Rényi Insititute of Mathematics, Budapest, Hungary Fötvöö Loránd University, Budapest, Hungary fcsadrian, koszpe, matazang, gergopool, daniel)@renyi. hu
Abstract
We employ a toolset dubbed Dr. Frankenstein to analyse the similarity of representations in deep neural networks. With this toolset, we aim match the activations on given layers of two trained neural networks by joining them with stitching layer. We demonstrate that the inner representations emerging in deep convolutional neural networks with the same architecture but different initializations can be matched with surprisingly high degree of accuracy even with a single, affine stitching layer. We choose the stitching layer from several possible classes of linear transformations and investigate their performance and properties. The task of matching representations is closely related to notions similarity. Using this toolset, also provide novel viewpoint the current line of research regarding similarity indices of neural network representations: the perspective of the performance on task.
Introduction
central topic the analysis of deep neural networks is the investigation of the learned representa- tions. A good understanding of the features learned on inner layers provides means to analyse and advance deep learning systems. fundamental question representations similar?" We contribute to this line of research by introducing novel approach to study the above question. The key idea of our work is to ask the following, somewhat different question: "in what way two representations similar, once know that they similar?" In this paper, we present a conceptual framework and a methodology that allows to meaningfully pose and study this question. For this purpose, will distinguish two approaches define similarity: representational similarity and functional similarity.
this line of inquiry "when are two
Representational similarity Representational similarity notions define similarity via computing statistics on two different data embeddings that capture appropriate geometrical or statistical prop- erties. There exist several statistical measures, [Raghu al. 2017 MMoroos et al. 2018 Kornblith et al. 2019 each which serves as a different notion of similarity. These measures have been used
successfully obtain valuable insights about the inner workings of deep neural networks [Nguyen
et al. 2021 Mirzadeh et al. 2021 Neyshabur et al. 2020 Wu et al. 2020).
While representational similarity concerns the data embeddings, in contrast, functional similarity concerns the functions that produce and process these embeddings, i.e., parts of the function compositions that the neural networks realize. With these elements, we can ask new types of questions that we were not able to ask using solely the data embeddings. For example, we can ask the following: "can network B achieve its task using the representations of network A?"
Functional similarity
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Our paper focuses on this specific theme. In particular, we investigate this by taking the activations of network A at given layer, transform it with an affine map, then use the result as an input of the same layer in network B. In other words, we stitch together the two networks with an affine stitching layer.
This technique first appeared in Lnne and Vedaldi [2019] to study the invariance and equivariance properties of convolutional networks. Evaluating the performance of this combined network provides an alternative viewpoint on similarity of representations, the viewpoint of functional similarity.
We investigate this novel perspective of reflecting on representational similarity through the lens of functional similarity. A brief outline of this work is the following:
After detailed introduction to model stitching (Section 3 and presenting two types of stitching methods (Section 4). we empirically demonstrate that trained convolutional networks with the same architecture but different initializations can be stitched together, even with a single, affine stitching layer in many cases without significant performance loss (Section 55. We will refer to this compatibility property as the 'matchability' of representations.
Observing the matchability of representations as a quite robust property of common vision models, we have a wide range of examples in our hands when we know that representations are functionally similar. This permits us to study the relation between representational similarity measures and functional similarity. In our experiments, we show that the values of similarity indices are not necessarily indicative of task performance on a stitched network, and perhaps more surprisingly. perforrerormnce stitchings can have differences in the values of similarity indices such as Centered Kernel Alignment (CKA) [Cortes et al.] 2012 Kornblith et al. 2019]. This also reflects on phenomena experienced by the "end-users' of these indices. For instance, in the context of continual learning Mirzadeh et al. [2021] observe that CKA remains constant, while the accuracy on previous tasks drops drastically (Section
Constraining the space of transformations in the stitching layer provides means to analyse how the information organised in the representations. As an example of this probing methodology, we study bottleneck stitching layers, and show that when doing SVD in the representational space, the magnitudes of the principal components are not in direct correspondence with the information content of the latent directions (Section Zp. As SVD is commonly used with embeddings, and there are representational similarity measures, for example SVCCA [Raghu al.) 2017], that use SVD as an ingredient, these results might be of interest to both theorists and practitioners.
The transformation matrices of well-performing stitching layers capture how representations are functionally similar. This provides the opportunity to empirically investigate "in what way are two representations similar, once we know that they are similar?". As a first step on this avenue, we present results regarding the uniqueness and sparsity properties of the stitching matrices (Section 8).
2 Related work
Matching representations The idea of 1x1 convolution stitching layers connecting the lower half of network with the upper half of another network first appears in Lene and Vedaldi [2019]. They demonstrate that such hybrid Frranken-CNNS can approach the accuracy of the original models in many cases. For shallower layers, this phenomenon can appear even when the first network is trained on a different task, suggesting that lower level representations are sufficiently generic.
Mapping layers as introduced by Li et al. 2016) are single layer linear 1xl convolutional networks mapping between two representations. They do not incorporate information from the second half of the network. The authors train mapping layers with LI regularized MSE, given their main goal of identifying sparse mappings between the filters of two different networks. As Li al. (0166] notes, learning mapping layers can get stuck at poor local minima, even without LI regularization. the shallowest layers of the networks they
This forces the authors to restrict their experiments inspect. We manage to overcome the issue of poor local optima by initializing from the optimal least squares matching (Section 8. and as our experiments demonstrate, this alleviates the issue, resulting in stitched networks that offen achieve the performance of their constituents.
Concurrently and independently of our work, [ansal et al. [021] also apply model stitching to investigate similarities between deep learning models. They observe that for a given task, even very different architectures can be successfully stitched, e.g., a transformer with residual network.
Cooee is available the project website: httph::////tbtt 1y/ssmmmmmmmmm http:/itimllriiiyooooora and-matching

---

Similarity indices Canonical Correlation Analysis (CCA) provides a statistical measure of similar- ity. CCA itself has a tendency to give too much emphasis to the behavior of mawnitude high-noise latent directions. Similarity indices were designed to make CCA more robust in this sense, namely Singular Value CCA (Raghu et al. 2017] and Projection Weighted CCA [Morcos et al. 2018].
The current state of the art in measuring similarity is Centered Kernel Alignment (CKA) (Cristianini al. 2001 Cortes al. 2012 Kornblith al. 2019]. Kornblith et al. (2019) conducted comparative study of the measures CCA, SVCCA, PWCCA as similarity indices, and they observed that the other similarity indices do not succeed in identifying similar layers of identical networks trained from different initializations, whereas CKA manages to do so, therefore in this paper we chose to focus on the latter. Kornblith et al. [2019] also consider nonlinear CKA kernels, but as they demonstrate this does not lead to meaningful improvements, so we focus only on the linear variant (also commonly known as RV coefficient Robert and Escoufier (1976). () in line with other literature employing CKA
similarity Nguyen et al. 202 Mirzadeh et al. 2021 Neyshabur et al., 2020. Wu et al. 2020].
Functional similarity The general framework of knowledge distillation [Hinton al. 2015
Romero et al. 2014] can be seen as fostering end-to-end functional similarity of student net- work to a teacher network. This relation and the versatile utilisation of this technique underpins the importance of the notion of functional similarity. Yosinski et al. (2014) studies transfer learning from a related perspective.
3 Introducing Dr. Frankenstein
3.1 Preliminaries and notation
Let fo : X' + y denote the input-output function of feedforward artificial neural network with m E N. layers. thus, fe = fm o fi, where X' is the input space, y is the output space, fi AA,11 A, are maps between activation spaces A; and A; for i [m]] with An = X' and are the parameters of the network. We consider models that are trained on dataset D = {(;; v/ )=
supervised manner with inputs I; E and labels y: E V, where n E N is the dataset size.
The central tool of our investigation involves splitting up the network at a given layer to a representa- tion map and a task map. The representation map layer L is a function R2 X + Ar that maps each data point z; to its activation vector d, at layer L:
(x,) fu
fi ds,L.
where Ar the activation space at layer L. The activation vectors for the dataset D at layer are simply denoted by A,, (04.r.)== The task map at layer I, is function T, Ar. - y that maps an activation vector at layer T. to the final output of the network:
(a(,,)) = fm o ....
.... o f1+1((,,,))
Thus, the input-output map of the network fa is simply the composition fe = Tú Rí :X' - y for all = 1,...,,.. m. We will omit the index of the layer when this does not hurt clarity.
3.2 The Frankenstein learning task
Definition 3.1 (Frankenstein network). Fix two (pretrained) neural networks fe and fo. We will call a transformation S Ae,L - As.M between the two activation spaces at layers and M stitching layer. Given stitching layer S, the Frankenstein or stitched network corresponding to S is the composition F Fs
Fxx)= = To,M S Ro,
where To.M M. is the lask map at layer M for the network J and Ra, is the representation map at layer L for network So.
The stitching layer thus realizes a correspondence between the representations of two neural net- works: it transforms the activations of one network a particular layer to be a suitable input to the corresponding layer in the other network. For better readability, we will often call So as Model I and Ja as Model 2, and index the corresponding data accordingly (Si = fo. f2 = Se. etc.). In the experiments of this paper, Model and Model have the same architecture and L = M.
3

---

Netwok. /g
4
Lapo
Laper
Laper
Laper 4
Layer
Died
naschin Layer
Uipe
Layers
Layo-s
Laner
Lapprr
Layor
Laporr
Layor
Lapore
Dwatert
TEEE
CORK
(a) Direct matching of representations.
(b) Task loss matching of representations.
Figure 1: Illustrating two types of matching approaches. (a) Direct matching: first align the representations using only the representations themselves, and use the forward path the task map of Model 2 - with the transformed activations - only to evaluate the performance of the stitched model. (b) Task loss matching: the forward path from the input starts on Model then through the stitching layer it continues on Model 2, then task loss gradients are backpropagated to update the weights of the stitching layer.
Definition 3.2 (Frankenstein learning task), Given two pretrained neural networks fo and fer task given by a labeled dataset D {(1g, y4) E X y C: y x y + R. we will call the Frankenstein learning task at layer L the task of finding the stitching layer S Ao.L + Ao.LL which minimizes the total value of the loss C the dataset D.
1..... ,,} and loss function
Frankenstein is an evaluation framework The Frankenstein learning task primarily provides an evaluation framework for matching representations, and it does not tie our hands on how the transformation function S is produced. As an evaluation metric, value of the Frankenstein learning task on an evaluation set, or any other appropriate metric (e.g.. accuracy in the case of classification tasks).
either use the mean loss
Stitching convolutional representations Our toolset is generic, but here will only consider convolutional architectures. Thus, the activation spaces have the structure of rank-3 tensors Rexxxe where and h are the spatial width and height and is the number of feature maps. We will only consider linear stitching layers of the form M : IR + Re (1 x 1 convolutional layers). When formulating least squares problems on many activation tensors, we always mean the least squares problem in IR which in practice means reshaping the x w x h x tensor to nuch x c matrix.
4 Methods to solve the Frankenstein learning task
Here we line up several possibilities to solve the Frankenstein learning task presented in the previous section. We distinguish two approaches: (1) Direct matching: utilize only the outputs of represen- tation maps; (2) Task loss matching: utilize the loss of the Frankenstein learning task. Figure illustrates the difference between the two approaches schematically.
4.1 Task loss matching
From the exposition of the Frankenstein learning task, a training method naturally follows: utilize the loss function in the Frankenstein learning task itself, and train appropriately parametrised stitching layer with backpropagation (leaving the other parts of the Frankenstein network fixed).
We use the outputs of Model 2 as a soft label [Hinton et al. 2015 and define the task loss with cross-entropy. Another meaningful option would be to use the original training objective (the one that was used for the training of Model 1 and Model 2, e.g., the one-hot labels in classification task). Both are reasonable choices as tasks, one corresponding to the stitched network imitating Model 2, the other to solving the task itself. We have experimented with both losses and found very high correlation, with no difference regarding our main observations. We often present performance in terms of relative accuracy compared to Model 2, because it is casier to interpret than cross-entropy.

---

4.2 Direct matching methods
Due to their close connection to similarity indices of representations, we also consider methods that arrive at the transformation S by using only the representations themselves, namely, utilizing only the outputs A, B E R=xP of the representation maps of Model and Model 2, respectively.
Least squares methods Given A, B E _uxP, and class of transformations C. C Horm (OT, R?), we will consider least squares problems of the following kind: find an optimal M, such that
AMM B
min AM - BBl
(1)
MEC
We consider three variants of this problem: 1. arbitrary linear maps: C Hom(R2 R2), 2. orthogonal transformations: Opp). 3. linear maps in Hom (?) of rank most k: C Σg.
In each case, there is a closed form expression relying on singular value decomposition, For C Hom(R R.)) the minimal value of ①T) is obtained through an orthogonal projection in the space of matrices Penrose 1956
M, MLS B,
where A denotes the Moore-Penrose pseudoinverse of A. We will refer to this MLS as the least squares matching. For results of this type, Section
In the case = O(p). problem [ is also known as the orthogonal Procrustes problem, see e.g. Golub and Van Loan 2013]. If A USV is singular value decomposition, then the optimal orthogonal map is M. UVT.
Finally, the case C = Σg is also known as reduced rank regression Izenman 1975] In terms of the SVD of AM,s USVT, the optimal rank map M,, given by M, Mrs Ve V.. where 1 consists of the first & columns of V. For reduced rank matching results, see Section 7
Sparse matchings It is known [Wang et 2018] that representations are not fully local, that is, matching cannot be achieved by permutation matrices. High quality sparse matchings between neurons can be interpreted imply "slightly distributed" representations [Li et al. 2016].
To achieve sparsity, a natural approach is to add the regularization term ||M] = Σ may to the loss
function, where M is the transformation matrix. Then one consider the L1 regularized versions of 1) task loss matching, and 2) least squares matching (Lasso regression, as in Li et al. (016 )
B) |AM - B| + a- ||M
where A, B denote the activations of the two models. See Section for these experiments.
5 Matching representations in deep convolutional networks
We now continue our exposition by presenting series of experiments regarding matching neural representations. Our results in this section can be summarized in the following statement:
Neural representations arising on a given layer of convolutional networks that share the same architecture but differ in initialization can be matched with a single affine stitching layer, achieving close to original performance on the stitched network.
5.1 Experiment 1: Matching with least squares and task loss matching
In this experiment, compare the performance of direct matching and task loss matching. We take two networks identical architectures, but trained from different initializations. We perform direct matching well task loss matching from Model Model 2. We then compare the performance of the stitched network to the performance of Model
We conduct experiments on three different convolutional architectures: simple, 10-layer convnet called Tiny-10 (used in Kornblith et al. (2019), on a ResNet-20 [He et al. 2016), and on the Inception VI network (Szegedy et al. 2015]. These networks are rather standard and represent different types of architectures. Tiny-10 and ResNet-20 are trained and evaluated on CIFAR-10, Inception VI on the 40-label CelebA task. Further training details follow common practices (sce Appendix A.3 A.3)
5

---

MP Satten:
W.B Suttalt Cod
W-P Satiael Chil WID
UnSe Lasr Squans CoEs-INoe
uever Lasr Solant OOEEUITEE
34-
Uapert
LayetLa
Lapeiia
Sapeaa
Sareakt
Kapert
Super
Gapea
Mondae
uxpl
(a) Results with ResNet-20..
(b) Results with Tiny10.
(c) Results with Inception V1.
Figure 2: Performance of Frankenstein networks stitched together specific layers (horizontal axis) measured in relative accuracy and cross-entropy (left and right vertical axes, respectively). This experiment compares two methods: task loss matching and linear least squares matching (a direct matching method). Results are averages of 10 runs, the bands represent standard deviations. As baseline values for cross-entropy, we provide average cross-entropies between Model and model stitched without transformation: 10: 6.99 + 1.96, ResNet-20: 4.69 + 2.91, Inception 0.36 + 0.08 (averaged across layers).
0.902
0.903
0.003
0.75
With 54.
Wan
Relative Medel
Figure 3: Relative accuracy values with ResNet-20 nets of different width. The horizontal axis shows the width multiplier compared to the baseline network. Plots correspond to different layers with names denoted the top. Averages of 10 runs, bands show standard deviations.
Figure 2 summarizes the results: task loss matching achieves near perfect accuracy ResNet-20, and also succeeds in aligning representations on Tiny-10 without collapsing accuracy. While the direct matching significantly underperforms the task loss matching and collapses performance several layers, it is still remarkable that direct matching performs fairly well in many cases: e.g., on the layers between Layerl.0 and Layer2.1 on ResNet-20 almost every case Tiny-10.
Further results We have experimented with networks with batch normalization and without batch normalization, and networks trained with different optimizers (SGD, Adam). Still, we did not encounter any cases where there were unmatchable instances. The worst performing stitchings were after the first convolutional layer on ResNet-20, still achieving 80% relative accuracy. While this method is not universal on arbitrary architectures, this does not modify the observation that 'matchability' of representations is quite robust property of common vision models.
In the case of task loss matching, we found the most stable results when we initialized the training of the stitching layer from the least squares matching (2). See Section further details on how the initialization influences performance. We also evaluated Procrustes matching described in Section 4.22 and found that on later layers, it achieves the same level of accuracy as linear least squares matching.
As detailed in Section C.2 of the Appendix, accuracy drop during stitching can be surprisingly low even when we stitch networks trained on different datasets; in our case, ImageNet and CelebA.
5.2 Experiment 2: Networks of different width
This experiment focuses on the effect of network width on stitching performance. Matching happens with cross-entropy to Model output as the task loss. The stitching initialized to the optimal least squares matching. Performance is measured on the original classification task, relative to Model 2 performance. sigure the results on ResNet-20 with various widths. The relative accuracy for 1x-width is already very high, but clear trend of performance increasing with width is still visible.

---

Similarity indices and matching representations
We now turn our attention the relation of similarity and matchability of representations. Finding an appropriate similarity index for neural representations is an active area of research [Gretton et al. 2005 Raghu al., 2017 Morcos et al., 2018, Kornblith et al., 2019]. In the pursuit suitable notion, one seeks to capture the right invariance and equivariance properties to provide a tool with theoretical guarantees and practical applicability. Our brief summarizing statement this section acts warning for the end-users of statistical similarity indices:
By their nature, similarity indices depend the task only indirectly, and their values are not necessarily indicative of task performance on stitched network.
6.1 Similarity indices
Given two sets of activations A A2 IR",, similarity index <A, B) [0, 1), measures the similarity of the two data representations, i.e. it a statistic on the observation space & .A x A + [0. 1]. We now briefly review the similarity indices under consideration. First, the least squares matching MLs defined earlier can be used to define a statistic that can be used as similarity index (Kornblith et al. 2019]:
B (b,) in the activation spaces a, E A1 B3,
Definition 6.1. The coefficient of determination of the best fit given in terms of the optimal least squares matching MLS- A as:
RE (A, = |AMss BI
BB
Let k, be positive definite kernels on space X' and let Kij k( mz) and T.j = [( 2z)) be the Gram matrices of a finite sample z, E X, i == I.... m drawn from distribution on X. The (empirical) alignment of the Gram matrices K, L was defined by Cristianini et al. [200] as
(K,L)F L)F
KA(K, L) = |||| |
The importance of centering the kernels in practical applications was highlighted Cortts et al. 2012). Given Gram matrix K, denote the centered Gram matrix by K" CKC, where C' ==
1.1" is the centering matrix size m. The Centered Kernel Alignment of the Gram matrices K, L is defined [Cortes et al. 2012 as CKA(K, L) = KA(K" LT). If the kernels are linear on their respective feature spaces, and if X, Y. are samples of the distribution in these feature spaces, then CKA(Kx, Ly) KA(Kxc. Ly using that K K Kxc- In particular, in the of linear kernels CKA is also referred to as the RV coefficient [Robert and Escoufier) 1976]. Furthermore, X, Y are the centered matrices representing the samples of the distribution the feature space, CKA can be given by the following definition:
Definition 6.2. The empirical (linear) Centered Kernel Alignment or RV coefficient of the samples X, Y centered in feature space is
T |XTX| F- |YTY|F
RV CKA(KX,
In our context, the probability distribution describes the data distribution in X, the feature spaces are the activation spaces and the feature maps the representation maps R X A.
As a measure of similarity of representations, CKA was introduced and used successfully in Kornblith et al. 2019 They also considered other similarity indices such as CCA, SVCCA, PWCCA [Raghu ctt al. 2017 Morcos et al. 22018) in their experiments, but they found that the other similarity indices do not succeed in identifying similar layers of identical networks trained from different initializations, whereas CKA manages to do so. For this reason, we chose to focus our attention on CKA.
6.2
Experiment 3: Similarity indices and task performance
In this set of experiments, we monitor the CKA, RE. CCA and SVCCA indices throughout the training process of the stitching layer trained with task loss matching. We measure these values

---

OCA
0.94
0.962
D6K Kun
0.33
Modsl
0.565
Cross
0.8
0.92
Eress meroa Ne
0.51
0.23
0.6
0.355
0.4
0.84
CRA
Cres-entropy to Mode 2
3.88
0
0.
Reistiue accuraty Model
2500
2020
400
603
Seratio
Earation
Earation
Figure 4: CKA, R and cross-entropy values over the training iterations of task loss matching started from the optimal least squares matching. Tiny-10 network, stitching at Layer 3. Bold lines are averages 10 runs, with bands representing standard deviations; dotted and dashed lines correspond to two specific runs to show the individual characteristics of the training of the stitching layer. Further experiment details are in Appendix A.55 A.5
Figure 5: CKA and performance measures throughout the train- ing for a Frankenstein task with a loss that explicitly penalizes CKA value while maintaining the model performance. Stitching at Layer 2.0 of 3-wide ResNet.
between Model 2 activations and the transformed activations of Model 1. Figure 4 shows the results for CKA and R We observe that CKA has a large variance from the first step of the training process, and it is decreasing while the performance of the stitched model is increasing.
This experiment also highlights that Rf. can correlate inversely with the performance: as the training proceeds with decreasing cross-entropy, the REa values also decrease. Indeed, since we initialized the task loss training from the optimal least squares matching, the distances naturally must increase under any training process. However, this highlights the fact that the least squares matching is ignorant of the decision boundaries in a neural network.
CCA is invariant to invertible linear transformations, hence it ignores our linear stitching transforma- tion during the training. SVCCA also performs CCA on truncated singular value decompositions of its inputs. Because its value only depends on the SVD step, it performs quite big jumps during the training, but on average it does not correlate with the task performance. See further results and experimental details in Appendix A.5
6.3 Experiment 4: Low CKA value with high accuracy
We now show that it is easy to align representations in a way that results in high task performance while the CKA value at the alignment is small. The setup is the following: we take one trained model and stitch with itself. utilizing a linear stitching layer initialised with the identity mapping. With this initialization we essentially reproduce the original model, thus, this trivially results in Frankenstein network with the same performance as the original model. We train this stitching layer with an additional loss term that penalizes the CKA value at the stitching, meanwhile we maintain the task performance with the usual cross-entropy term. We use the CKA value itself as the penalty, as it is differentiable by definition and thus suitable for gradient-based optimization.
Figure shows the outcome of such training process. We find that the CKA value drops by large amount (achieving lower than 0.7), whereas the accuracy remains essentially unchanged.
As bascline, we provide the CKA values between representations of the transformed activations of Model and Model 2 averaged over different network pairs and all layers. ResNet: 0.922 + 0. 0.098, Tiny-10: 0.883 + 0.050. For further baseline CKA values we also refer to Kornblith et al. 2019].
6.4 Discussion
Notions of representational similarity and functional similarity have different motivations, and thus, aim to capture different invariances. Part of these invariances could be compatible or incompatible. For example, assuming that a similarity index is invariant under given class C of transformations, it is not surprising that the high value of similarity index s(A, B) does not indicate the relative performance of the activations and B: indeed, given a set of activations A, ((AF, A) = for any F E C, while the task loss of the transformed activations AF typically varies greatly. On the other hand, is somewhat surprising that high-performance matchings can have significant drop in the values of meaningful similarity indices such CKA.
8

---

Layer
Layer
Laye2
Leyat Nasdoon
Hankin
8.0
2.00
14
3339
132
With Task bis Reduced Rank
With Task loss
Random
fandor
0.2
Roduced Rank
Procrustes
0.0
Rolative Accuracy Model
oanc sttchiig makni
Rant otsttching marts:
Figure 6: Relative accuracies by initialization type.
Figure 7: The performance of the low-rank analogues of least squares and task loss matchings in terms of relative accuracy.
The inadequateness of CKA as a direct matching method CKA is not directly suitable as an objective for direct matching, since is invariant to orthogonal transformations. A stitching with an optimal CKA value be arbitrarily transformed via orthogonal transformations without affecting CKA, but most of these equivalent solutions are unsuitable for the matching task as trained models not invariant for arbitrary rotations in the activation spaces.
7 Low rank representations a gap between SVD and task loss matching
In this experiment, we compare the effectiveness of the low-rank analogues of the above presented least squares matching and task loss matching. More precisely, we apply SVD directly in the activation space to obtain the low-rank least squares matching, and use a bottleneck stitching layer to realise low-rank matching with the task loss.
Figure depicts the results for two selected layers of Tiny-10 with prescribed ranks 4, 8, and 16. (We observed similar results for the other layers, and also for ResNet-20.) is clear that task loss matching outperforms least squares matching in the activation space when the rank of the transformation is fixed. This suggests that the objective for the low-rank approximation of explaining most of the variance in the activation space (i.e., the objective of SVD) is not satisfactory to attain the highest possible performance with prescribed rank. In other words, the magniudes of the principal components are not in direct correspondence with the information content of the latent directions. Ultimately, this suggests that similarity indices that use SVD directly in the activation space either as preprocessing step or as another key element in the computation might fail to capture some desired aspects of the representation. Details and figures for the other layers are in Appendix A
Structure of stitching layers: uniqueness, mode connectivity, sparsity
The previous experiments show that the stitching layer can be trained to achieve the same performance as the original Model 2. What is the structure of the obtained stitching matrix? In this chapter, we explore the question of sensitivity with respect initialization, uniqueness properties of the stitching layer and the relationship of direct and task loss matching with sparsity.
Experiment 5: Dependence of accuracy initializations Are stitching layers obtained from different initializations the same? A crude measure of this is the performance of the stitching layer on the stitched network. In this experiment, different initializations. We train the stitching layer on task loss from 50 random and 50 optimal least squares initializations, and compare their performance. We found that in most cases the stitched networks achieve the same order of accuracy, although not always; the training process might get stuck poor local minima. However, the optimal least squares matching initialization performs consistently well. Figure showss a histogram of the results. For further details, see Appendix A
fix two identical network architectures trained from
Experiment 6: Linear mode connectivity of transformation matrices We also investigated the uniqueness properties of the stitching layer by examining the linear mode connectivity of the found transformations. particular, on a given layer of two trained copies T1ny-10, we trained stitching transformations from different random initializations. We only kept the transformations achieving a relative accuracy of at least 80% there no reason to expect mode connectivity properties where there is a large difference in relative accuracy. We then evaluated the relative accuracy of the stitched network with the transformation matrix My XM, + (1 - X)M2 for pairs of transformation matrices Mj and M2, and different values of X E [0, 1]. The results showed that mode connectivity

---

aCC MAEAS
bal SLOEN
D-ecet Mazching
Toalsttr Modkll
meistrg [Jinst (Nrettratteve
Figure 8: Comparison stitching matrices and the relative accuracy of Ll-regularized direct and task loss matching. To achieve sparsity, we set every entry of the stitching matrix to zero when the absolute value of the entry is below the threshold 10-1, Sparsity is the ratio between the zero elements and the total number of elements in the stitching matrix. Relative accuracies with zero sparsity correspond to the unregularised baseline. Left: Examples of stitching matrices with different sparsity. Right: The degradation of relative accuracy with respect to increasing sparsity is significantly less steep with task loss training. For more results, see Appendix A.9
holds on most layers, i.e. the convex combination of the stitching layers perform similarly. However, on the first three layers, we noticed an unforeseen lack of mode connectivity, even though the relative accuracies of the corresponding transformations were high. For further details, see Appendix A.8
Experiment 7: Sparsity - direct vs. task loss matching How does the sparsity of the stitching layer influence the performance of the stitched network? In this experiment, we achieve sparsity by adding an Ll-regularizing term to the training as detailed in Section 4.2 similarly to the experiments in Li et al. [2016). We perform both an I. L.-rreularized task loss matching as well as direct loss matching with loss term (3). To obtain a truly sparse stitching matrix we set the elements to zero below threshold. Sparsity is then computed as the ratio between the zero elements and the total number of elements in the matrix. Figure shows the dependence of relative accuracy on sparsity compared for task loss and direct matching well sample of the stitching matrices. As Figure & shows, at around 80-90 % sparsity, the relative accuracy direct matching suffers a drastic drop, however task loss matching still manages to find meaningful aligmment of the neurons using additional information via the task loss. These experiments show that even though it is possible to achieve certain degree of sparse alignment of the neurons, the relationship between two representations is typically not of combinatorial nature, and much can be gained by considering more complex relationships between the two layers. Furthermore, the difference in the relative accuracy of direct and task loss matching is notable for higher values of sparsity. For further details, see Appendix A.9
9 Limitations
The performance of the stitched solution is inherently just a lower bound due to possible local minima. The current study is restricted to well-establishe convolutional architectures. Our observations network width is analysed only in isolation.
10 Conclusions
We studied similarity of neural network representations in conjunction with the notion of matchability. We demonstrated the matchability of representations in convolutional networks through several experiments and analysed the properties of the stitching transformations with respect to the task performance. Regarding similarity notions, we provided novel perspective which incorporates the performance on a task; a perspective which is yet unexplored the field. We pointed out weaknesses of popular similarity notions that end-users might encounter.
Broader impact
Our research is of foundational nature and advances machine learning at fairly general level. While the authors do not foresee that the work herein will have immediate adverse ethical or societal consequences, such analysis of representation learning could help malicious user avoid detection that is based on representational similarity; on the other hand, it could move forward the field towards more transparent and interpretable machine learning models.
10

---

Acknowledgements
This work was supported by the European Union, co-financed by the European Social Fund (EFOP- the Hungarian National Excellence and by the Hungarian Ministry of Innovation and Technology NRDI Office within the framework of the Artificial Intelligence National Laboratory Program. We thank the anonymous reviewers for their many valuable suggestions.
-0--VEKOP-----017-000000
References
Yamini Bansal, Preetum Nakkiran, and Boaz Barak. Revisiting model stitching to compare neural representations. Advances Neural Information Processing Systems, 35, 2021.
Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kernels based on centered aligmment. The Journal of Machine Learning Research, 13(1):795 2012.
Nello Cristianini, John Shalor André Elissefff, and Jaz Kandola. On kernel-target alignment. In Advances Neural Information Processing Systems 14. Citeseer, 2001.
Gene H. Golub and Charles F. Van Loan. Matrix computations. Johns Hopkins Studies in the Mathematical Sciences. Johns Hopkins University Press, Baltimore, MD, fourth edition, 2013.
ISBN 978-1-4114-079444 1-4214-0794-9; 978-1-4214-0859.0.
Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with Hilbert-Schhiid norms. In Sanjay Jain, Hans UIrich Simon, and Etsuji Tomita, editors, Algorithmic Learning Theory, pages 63-77, Berlin, Heidelberg, 2005. Springer Berlin Heidelberg. ISBN 978-3-540-31696-1.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016. doi: 10.1 10.1109/CPP..2016.0.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network, 2015.
Alan Izenman. Reduced-rank regression for the multivariate linear model. Journal of Multivariate Analysis, 5:248-264, 06 1975. doi: 10. 100016/00472259X75590000
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference Machine Learning, pages 9519-3529 PMLR, 2019.
Alex Krizhevsky. Learning multiple layers of features from tiny images. University of Toronto, 05 2012.
Karel Lene and Andrea Vedaldi. Understanding image representations by measuring their equiv- ariance and equivalence. International Journal of Computer Vision, 127, 05 2019. doi: 10.1007/s11263-018 1098-y.
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent learning: Do different neural networks learn the same representations? In International Conference on Learning Representation (ICLR '1), 2016.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Dilan Gorur, Razvan Pascanu, and Hassan Ghasemzadeh. Linear mode connectivity in multitask and continual learning. In International Conference on Learning Representations, 2021. URL http::t::ppp/eeeeeeiiee
Ari S. Morcos. Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Montréal, Canada, pages $732-5741, 2018. URL wttphhttts///t / www.eediiinnnn..uiiipp neurips cc/ halppppppp2111 2018/ BaB////hhaaaaaaaa/eaaa/aaaaaaaaa haa.s///h.aaarodda77777111111110 htt.:/aa99aa.a0011-1---bbrreer.d

---

Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? CoRR, abs/2008. 1667, 2020. URL /tttltt://aaax/ /tttppps//axx.voorr/////////////
11687
Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. In International Conference on Learning Representations, 2021.
net/forum?id
KJNcAKY8tY4
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2(11):e7, 2017.
R. Penrose. On best approximation solutions of linear matrix equations. Proc. Cambridge Philos. Soc., 52:17-19, 1956. ISSN 0008-1981.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein, Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In NIPS, 2017.
Paul Robert and Yves Escoufier. A unifying tool for linear multivariate statistical methods: The RV-coefficient. Journal of the Royal Statistical Society: Series (Applied Statistics), 25(3): 257-265, 1976.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprinl arXiv:1412.688 2014.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps, 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1-9, 2015, doi: 10.100//CPRR.2055.7988
Liwei Wang. Lunjia Hu, Jiayuan Gu, Yue Wu, Zhiqiang Hu, Kun He, and John Hopcroft. Towards understanding learning representations: To what extent do different neural networks learn the same representation. In Proceedings the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, page 9607-9616, Red Hook, NY, USA, 2018. Curran Associates Inc.
John Wu, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. Sim- ilarity analysis of contextual word representation models. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguisties, pages 4638 4655, Online, July 2020. Association for Computational Linguistics. doi: 10. 16.1166//112220.alllmm
URL
https::/www. awwweb..cwbbb..m org/ httg//nntology///0 2020 acl- main.. 422
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? arXiv preprint arXiv:1411.1799) 2014.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Yes, briefly in the Broader Impact section.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] We only reference theoretical results.
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] The source code is public. The instructions to reproduce the results (b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] The Appendix specifies the training details according to the standards of the field. The source code is public.
in the readme file.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] We report error bars for all our experiments.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] In Appendix AA]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] Code licenses are part of the repository.
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "vdorQ_8-aJl", "reviewer": "Reviewer_PmP3", "summary": "The paper presents a framework to evaluate the similarity of intermediate representations from two neural networks. Given two neural networks with the same architecture but trained with different weight initializations, the framework uses a stitching layer to search for a transformation between the intermediate representations of the two networks, such that the model performance on a given task is preserved. The paper further proposed two approaches to obtain such transformations. Under this framework, the author empirically observed that representations from a given layer of two convolutional networks can be matched.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. A core claim made by the paper is the empirical observation that under the proposed framework, representations from a given layer of two convolutional networks with the same architecture, but different weights can be matched. This claim is supported by the results in section 5 on three convolutional architectures (Tiny-10, ResNet-20, Inception-V1) on two tasks (CIFAR-10 and CelebA). However, it is not obvious whether the matching method is generic as claimed on line 100, even to other convolutional architectures. It is also unclear how architecture, tasks, training procedure can affect the observation. More explanation and descriptions should be made to this part. \n\n2. I found the paper a bit hard to follow, especially the experiment section. As the proposed toolset is expected to provide an evaluation framework on representation matching, I would expect to see how the similarity metrics, namely the loss or accuracy when training with the stitching layer, is correlated with 1) matchability of the representations and 2) with the “ground truth” representation similarity. Yet the experiment section 5 is extensively about comparing different methods to obtain the stitching layer, while section 6 is only comparing with existing metrics (similarity indices).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "none", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "Jr_TTl1D_8z", "reviewer": "Reviewer_yiYb", "summary": "The paper proposes a new tool using a stitching layer to analyze the representational similarity between two deep neural networks (DNN). The idea behind the stitching layer is that it connects one frozen network to another via a parametrized layer that is optimized using the soft labels obtained from one of the frozen networks. The intuition is that if representations of two different networks are highly similar the final performance obtained using the stitching layer should be close enough to the original model's performance.   \n\nThe key experiments demonstrate the following:\n1. Using the stitching layer DNNs with the same architecture but different initialization can be matched almost perfectly\n2. Standard similarity measures in the literature e.g. Centered Kernel Alignment(CKA) are not indicative of final task performance using stitching layer. Even with a low value of CKA, high performance can be obtained on the target task using stitching layers.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**UPDATE: After reading the author's response and planned updates, I believe the paper will be valuable to researchers interested in applications of neural network similarity in functional usecases. Therefore I have updated my score to 7**\n\n**Originality:** The paper introduces the idea of stitching layers to analyze similarities between neural network representations. The idea is derivative of [9] where they used a stitching layer to inspect equivalence between DNNs trained on different tasks. Here, the authors show results on comparing DNNs trained using different initializations.\n\nThe improvement on the original idea from [9] by using the optimal least-squares solution as initialization of the stitching layer is new. As shown in Figure 6, it improves the matching and therefore makes it possible to match even the deeper layers. \n\nThe comparison with CKA showing that the similarity measure is not indicative of task performance shares new insights into the limitations of these measures.\n\n**Quality:**\nThe submission seems technically sound with well-designed experiments to support the key claims. The experiments showing the importance of OLS initialization (Experiment 5) and comparison with CKA (Experiment 3 and 4) are well designed. Further, the experiments are performed on three different architectures and two tasks showing the generalizability of the claims.\n\nHowever, one important piece of information that is missing is which images were used to calculate CKA. If the number of images used to calculate CKA is less the results can be noisier, so it is crucial to provide these details.\n\nAnother weakness of the paper is that it shows the cases where the stitching layers work successfully. It might also be interesting to find out where they fail, for example, if two DNNs with the same architecture are trained on different tasks how successful is transformation using the stitching layer with respect to the depth of the network. How successful will the stitching be from a randomly initialized network?\n\n\n**Clarity:**\nThe paper is well-written with the methods clearly explained through equations and Figures. The results section is also organized well. \n\n**Significance:**\nIdentifying measures of similarity and their limitation is relevant to a wide set of applications such as transfer learning, multi-task learning, and interpreting what the network has learned. In this aspect, the paper introduces a new similarity measure based on task performance and shows the limitation of existing similarity measures.\n\nHowever, the paper doesn't show where stitching layers fail to provide as good performance as the original models. Such analysis might have revealed new directions about the potential applications of the proposed measure. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "HEGyAKMJ0S6", "reviewer": "Reviewer_j6kV", "summary": "The paper develops and applies a set of methods to determine the functional equivalence of architecturally-identical neural networks trained with different initializations. These methods involve taking the activations from a given layer l in one network, inputting them into a \"stitching layer\" (either a linear projection or a 1x1 convolutional layer), then taking the activations from the stitching layer and inputting them into layer l+1 in a second network. The stitching layer is optimized either to minimize the models' loss on a task, or to minimize the distance between its own outputs and the activations of the second network. The authors find that stitching layers can typically be learned that have minimal impact on model accuracy and loss. They also demonstrate some results that they claim undermine the utility of CKA, a popular representational similarity metric.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**UPDATE: I have increased my rating from a 3 to 7 in response to the authors' responses**\n\nI think the approach presented in this paper is clever and relevant, and could yield results that are worthy of publication. But I have a hard time feeling confident about the work in its current form. It’s difficult to assess whether the results are meaningful, both because of a lack of clear baselines and controls, and a lack of interpretation and explanation by the authors. The results are also limited to very small models and datasets. However, I am optimistic that my concerns can all be addressed through additional analyses and writing.\n\nI applaud the authors’ use of replicates and measures of variability in plots and experiments. This is necessary but disappointingly uncommon in empirical machine learning research.\n\nThe authors open with the question “When are two representations the same?”. This statement could be refined. “The same” is a binary assessment, but the paper seems interested in measures of similarity, and similarity is a continuous quantity. I think the present work attempts to dissociate functional similarity (e.g. changes in performance measures) and representational similarity, and the authors should focus on this distinction. I don’t mean to be pedantic—I think the work is well-motivated and the problem is relevant, but the paper opens with a framing of the problem that undersells it.\n\nOne general issue I have with this paper (which recurs in some of the comments below) is a lack of clear baselines for establishing whether some change in a quantity is meaningful. For example, what is the cross-entropy between model outputs at init? Or when the stitching method is applied to two networks at init? Or when the task loss/stitching layer method is applied, but the stitching layer is trained to minimize the *change* in the model’s outputs before vs. after stitching? All of these would help contextualize the current results.  \n\nThe authors only compare their results to CKA, choosing not to use CCA-based representational similarity measures because those “indices do not succeed in identifying similar layers of identical networks trained from different initializations” [L67-68]. Just because CCA-based methods are more sensitive to differences in initialization than CCA doesn’t mean they’re irrelevant to the present work. The present work develops methods to test whether differences in initialization are meaningful, so it seems sensible to me to include other representational similarity methods as a control.\n\nOne way the authors demonstrate the potential unreliability of CKA is by plotting raw CKA scores vs. cross-entropy (Figures 4 and 5). This is difficult to interpret without some baseline or context for what magnitude change in these quantities is meaningful. It also needs to be quantified. At the very least, compute a non-parametric correlation between these quantities. This extends to other representational similarity metrics, which I would encourage using (as per the previous comment). \n\nThe authors also demonstrate the unreliability of CKA by “stitching in” a linear layer initialized with an identity mapping, then training the layer using the task cross-entropy and a term to penalize the CKA value at the stitch. They find that they are able to maintain high accuracy while causing the CKA value to decrease. I am skeptical of this result for multiple reasons. Related to my previous comment, there is no context to determine whether this change in CKA value is meaningful (it even looks like accuracy decreases a small amount!), and the result is not quantified. Furthermore, CKA was specifically designed not to be invariant to all invertible linear transformations. Given the small changes in accuracy and cross-entropy, I suspect that the stitched-in layer simply learned a transformation to which the network is *less* sensitive than CKA. \n\nIt’s not clear to me what the authors mean when they state that they initialize from the optimal least squares matching (e.g. Line 180). Do they mean that the activations from Network 1 are first transformed using the linear projection obtained by solving a least squares mapping between Networks 1 and 2, and then the linearly-projected activations are fed into the convolutional layer?\n\nI think it might be helpful to attempt to dissociate the effect of the type of projection used in the stitching layer from the effect of the objective. Is it possible to solve the linear projection(s) using the task loss as the objective?\n\nIs it possible to use the objectives of representational similarity measures (e.g. CKA, CCA) for direct matching? If so, it could demonstrate the Frankenstein framework’s utility for benchmarking representational similarity measures.\n\nIt’s not clear to me how to interpret the sparsity results. The section is titled “​​direct vs. task loss matching”, but these conditions are not directly compared in the text, and it’s not immediately apparent to me that this comparison is meaningful. Is there a meaningful baseline that could be used here? Perhaps inserting a linear layer at the same depth as the stitching layer, then training the layer to maximize both sparsity and accuracy, and examining how accuracy changes as a function of sparsity? \n\nSome of the results are very interesting, but nearly all of them are presented without much interpretation. For example, what is the implication that Procrustes matching yields similar results as linear least squares matching for later layers but not for early layers [L175-177]? What is the implication of the lack of mode connectivity for stitching layer solutions in early layers?\n\nWhat is Layer 0 in the Resnet-20 (Figure 2) and why is performance so low when using it to stitch/match representations?\n\nWhat models are used in Figures 4-8?\n\nIn Figure 8b, the x-axis is labeled “sparsity”. How is this computed?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors only provide three sentences on the limitations of their work. I understand space is limited, but I think they could acknowledge the limited size of the datasets they use, and some of the instances in which their results were inconsistent and/or difficult to interpret.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "DIr1393wX7P", "reviewer": "Reviewer_AJnQ", "summary": "**EDIT: I have increased my score from 4 to 6 based on the authors' responses and their proposed edits**\n\nThe submission considers the representational similarity between two neural networks by fitting linear transforms between corresponding layers. The authors investigate \"direct mapping\", where the linear transformation is learned simply based on the layer activations, and \"task loss matching\", where the linear transformation is fit to optimize task performance of the \"Frankenstein\" network (layers 1–n from model 1, layers n–end from Model 2), when holding all other network parameters fixed. They observe that a single linear 'stitching layer' can successfully map between corresponding layers of two networks of identical architecture and training, differing only in their random initialization. The performance of the Frankenstein network depends on the architecture, the fitting method (direct fit vs task loss matching), and the layer at which the stitching is performed. When tested on Inception, there is no difference between layers or fitting method. The dependence on fitting method and layer are most pronounced for ResNet-20. They compare their results to Centered Kernel Alignment to highlight that low CKA similarity can be associated with high classification accuracy. Additional experiments attempt to characterize the stitching layers.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Some related work is missing. For example, the \"Frankenstein networks\" described here are quite similar to the \"selfer\" networks in Jason Yosinki's NeurIPS2014 paper: \"How transferable are features in deep neural networks?\". In that work, they similarly characterize the representations in neural networks by how interchangeable the heads/tails of the networks are, although without the addition of a stitching layer. Other work has also used layer-wise linear transforms to characterize representations, for example (Alain, G., & Bengio, Y. (2016). Understanding intermediate layers using linear classifier probes. ArXiv, 1610.01644v3). Romero 2014 used a least squares loss between the activations of two networks in work on distillation (Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., and Bengio, Y. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014.) The submission does reference the Kornblith 2019 paper, but doesn't make explicit how the Dr Frankenstein framework presented here differs from the Linear Regression condition tested in that work. The authors could also consider pointing to work in neuroscience which compares layers of neural networks to regions along a sensory hierarchy in the brain via linear matching (e.g. Yamins, D. L. K., & DiCarlo, J. J. (2016). Using goal-driven deep learning models to understand sensory cortex. Nature Neuroscience, 19(3), 356–365. https://doi.org/10.1038/nn.4244).\n\nI am unaware of another paper documenting the fact that one can learn a linear transformation between corresponding layers of two networks of identical architecture that preserves their performance. However, I disagree with the authors that this result is surprising or remarkable. They may want to elaborate on why this result is remarkable. \n\nThe submission focuses on the strengths of their work without carefully considering its weaknesses. The comparison to CKA feels somewhat incomplete. If the proposed framework is really intended to measure representational similarity, then it should be evaluated according to the same benchmark as CKA. Does Dr. Frankenstein find corresponding layers in two networks of identical architecture and training, differing only in their random initialization, to be most similar (relative to all other pair-wise comparisons)? Given its dependence on a linear transformation between layers, I suspect it will perform similarly to linear regression, which performed significantly worse than CKA on the layer matching task in Kornblith 2019. An honest discussion of the strengths and weaknesses would discuss when one might care more about task performance and when one might care more about the similarity of representational geometry.  Since CKA is only invariant to orthogonal transformations, not any invertible linear transformation, it is not surprising that the high performance matchings found here sometimes correspond to lower CKA scores. In fact, that is by design. Representational similarity is decidedly not just about task performance. One may obtain identical task performance with completely different features. One wants a similarity metric that is sensitive to those differences. However, I permit that there are certainly cases where task performance is what matters or where some insight about the network can be gleaned from analyses similar to the one presented here. But the submission does not present this as a tradeoff; they describe this characteristic of CKA purely as a weakness.\n\nThe submission is fairly well-written. However, I find that it somewhat over-complicates a relatively simple set of ideas. \n\nThe introduction could be improved by providing more motivation for why the authors chose to focus on matchability.\n\nThe most important contribution of this paper is the documentation of under what conditions one can arrive at a performance-preserving linear stitching layer between corresponding layers of two identical models (for the three models tested).  It is much less clear from the submission how the Dr. Frankenstein framework is useful or insightful as a method for measuring representational similarity.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors briefly discuss limitations but do not mention any potential negative societal impacts. No potential negative societal impacts readily come to mind. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.25"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper examines performance when an affine layer is used to “stitch” together layers from two neural networks, so that the hidden representations of one network are used as inputs to an intermediate layer of the other. Reviewers agreed that study of performance achievable via matching with either least squares or task loss was an important contribution. However, several reviewers expressed concern regarding the framing of the work, and in particular, the lack of clarity regarding the relationship between representational similarity methods and the proposed framework. At the urging of Reviewer j6kV, the authors proposed to change the introduction to describe the relationship between representational and functional similarity and to motivate the work from this perspective. Reviewers also raised several more technical questions. The authors provided a summary of the revisions they plan to make, and after reviewing this summary, all reviewers have recommended acceptance. The AC agrees that the paper, with the proposed revisions, provides a valuable perspective on representational and functional properties of neural networks.
Decision: Accept (Poster)