Revisiting Model Stitching to Compare Neural Representations
Authors: Yamini Bansal
Preetum Nakkiran
Boaz Barak
Venue: NeurIPS 2021 Poster (2021)
Keywords: representations
deep learning
neural network dynamics

Abstract: We revisit and extend model stitching (Lenc & Vedaldi 2015) as a methodology to study the internal representations of neural networks. Given two trained and frozen models $A$ and $B$, we consider a "stitched model" formed by connecting the bottom-layers of $A$ to the top-layers of $B$, with a simple trainable layer between them.  We argue that model stitching is a powerful and perhaps under-appreciated tool, which reveals aspects of representations that measures such as centered kernel alignment (CKA) cannot. Through extensive experiments, we use model stitching to obtain quantitative verifications for intuitive statements such as "good networks learn similar representations", by demonstrating that good networks of the same architecture, but trained in very different ways (eg: supervised vs. self-supervised learning), can be stitched to each other without drop in performance. We also give evidence for the intuition that "more is better" by showing that representations learnt with (1) more data, (2) bigger width, or (3) more training time can be "plugged in" to weaker models to improve performance. Finally, our experiments reveal a new structural property of SGD which we call "stitching connectivity", akin to mode-connectivity: typical minima reached by SGD are all "stitching-connected" to each other.


Revisiting Model Stitching to Compare Neural Representations
Yamini Bansal
Preetum Nakkiran
Boaz Barak Harvard University
Harvard University
Harvard University preetumècs. hhw.ardd edu
ybansalog harvard.
b@boazbarak. org
Abstract
We revisit and extend model stitching (Lenc & Vedaldi 2015) as methodology to study the internal representations of neural networks. Given two trained and frozen models A and B, we consider "stitched model" formed by connecting the bottom-layers of A to the top-layers of B, with a simple trainable layer between them. We argue that model stitching is powerful and perhaps under-appreciated tool, which reveals aspects of representations that measures such as centered kemel alignment (CKA) cannot. Through extensive experiments, we use model stitching to obtain quantitative verifications for intuitive statements such as "good networks learn similar representations", by demonstrating that good networks of the same architecture, but trained in very different ways (e.g.: supervised vs. self-supervised learning), can be stitched to each other without drop performance. We also give evidence for the intuition that "more is better" by showing that representations learnt with (1) more data, (2) bigger width, or (3) more training time can be "plugged in" to weaker models to improve performance. Finally, our experiments reveal new structural property of SGD which we call "stitching connectivity",
akin to conneccnncttivity: typical minima reached by SGD can all be stitched each other with minimal change in accuracy.
Introduction
The success of deep neural networks can, arguably, be attributed to the intermediate features or representations learnt by them [Rumelhart et al., 1985]. While neural networks are trained in end-to-end fashion with no explicit constraints on their intermediate representations, there is body of evidence that suggests that they learn rich representation of the data along the way [Goh et al., 2021, Olah et al., 2017]. However, theoretically we understand very little about how to formally characterize these representations, let alone why representation learning occurs. For instance, there are various ad-hoe pretraining methods [Chen et al., 2020a,b), that are purported to perform well by learning good representations, but is unclear which aspects of these methods (objective, training algorithm, architecture) are crucial for representation learning and these methods learn qualitatively different representations all.
Moreover, we have an incomplete understanding of the relations between different representations. Are all "good representations' essentially the same, is each representation "good" in its unique way? That is, even when we train good end-to-end models (i.c., small test loss), the internals of these models could potentially be very different from another. A priori, the training process could evolve in either one of the following extreme scenarios (see Figure I):
(1) In the "snowflakes" scenario, training with different initialization, architectures, and objectives (e.g., supervised vs self-supervised) will result in networks with very different internals, which are completely incompatible with one another. For example, even if we train two models with identical data, architecture, and task, but starting from two different initializations, we may end up at local
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

minima with very different properties (e.g., Liu et al. [2020]. If models are trained with different data (e.g., different samples), different architecture (e.g., different width), or different task (e.g.. self-supervised vs supervised) then they could end up being even more different from one another.
(2) In the "Anna Karenina" scenario', all successful models end up learning roughly the same internal representations. For example, all models for vision tasks will have internal representation corresponding to curve detectors, and models that better (for example, trained on more data, bigger, or trained for more time) will have better curve detectors.
"Snowflakes"
"Anna Karenina"
Exponentially many minima, separated by barriers
Low
High
level
level
Two extreme "cartoons" for training dynamics of neural networks. In the "snowflakes" scenario, there are exponentially many well-performing neural networks with highly diverging internals. In the "Anna Karenina" scenario all werl-perrorming networks end up learning similar representations, even if their initialization, architecture, data, and objectives differ. Image credits: Li al. [2018), Olah et al. [2017, 2020], Komarechka [2021].
Figure 1:
The "Anna Karenina" scenario implies the following predictions:
"All roads lead to Rome:" Successful models learned with different initializations, architectures, and tasks, should learn similar internal representations, and so if A and B are two such models, it should be possible to "plug in" the internals from A into B without significant loss in performance. See Figure 2A-B.
"More is b Better models trained using more data, bigger size, more compute, should learn better versions of the same internal representations. Hence is a more successful model than B, should be possible to "plug in" A's internals to B and obtain improved performance. See Figure 2C.
In this work, revisit the empirical methodology of ""model stitching" test the above predictions. Initially proposed by Lene and Vedaldi [2015], model stitching is natural way of "plugging in" the bottom layers of one network into the top layers of another network, thus forming stitched network (however care must be taken in the way it is performed, see Section 2). We show that model stitching has some unique advantages that make it more suitable for studying representations than representational similarity measures such as CKA [Kornblith et al., 2019] and SVCCA [Raghu et al., 2017]. Our work provides quantitative evidence for the intuition, shared by many practitioners, that the internals of neural networks often end up being very similar in certain sense, even when they are trained under different settings.
1.1 Summary Results
experimental tool. We establish model stitching way of studying the representations of neural networks. A version of model stitching was proposed in Lenc and Vedaldi [2015] to study the equivalence between representations. In this work, we argue that the idea behind model stitching is more powerful than has been appreciated: we analyze the benefits of model- stitching over other methods study representations, and we then use model-stithhing establish an number of intuitive properties, including new results on the properties of SGD.
Model stitching
'Alll happy families are alike: each unhappy family is unhappy in its own way." Leo Tolstoy, 1877. https://en. wikipedia, org/wiki. /tt///ikki//// Annaan Karaaaaaaannaaaaaa paremiinapprinciple

---

"All roads lead to Rome"
"More is better"
30-
35-
VIT
Restee-
LeM0
Susi
16K Gomples
25K camples
20 Modid
20-
sampres
Parzor Network
23
30-
10-
PyrtaCiey
Top model se
15-
Fandiu Nerwer
20-
Sellsspeeriied
100%
soncing
30-
100% intialization
100% 2nd
intialization
10-
100% EndRo-LEnd
Better regresentation improvess per formance
A
0.0
c.2
c'4
0.6
0.5 1.0
B
0.0
02
0.4
0.5
0.5
1
C
0.c
0.2
0.4
0.5
1.
Traction layers from bottorn model
Fraction layers from bottomm model
Fraction layer from bottomn model
Figure 2: Summary of main results (A) Various models trained on CIFAR-IU 10 identically except with different random initializations are "stitching connected". can be stitched at all layers with minimal performance drop (see Section 4). Stitching with a random bottom network shown for reference. (B) Models of the same architecture and similar test error, but trained on ImageNet with end-to-end supervised learning versus self- supervisd learning can be stitched with good performance (see Section 5). (C) Better representation obtained by training the network with more samples can be "plugged-in" with stitching to improve performance (see Section 6). In all figures, stitching penalty is the difference in error between the stitched model and the base top model.
In this paper, we use model stitching in the following way. Suppose we have a neural network A (which we'll think of as the 'top model") for some task with loss function L (c.g. the CIFAR-10 or ImageNet test error). Let X + Rd be a candidate "representation" function, which can come from the first (bottom-most) layers of some "bottom model" B. Our intuition is that r has better quality than the first E layers of network A if "swapping out" these layers with r will improve performance.
"Swapping out" is performed by introducing an additional trainable stitching layer to with A (defined more formally in Section 2). The stitching layers have very low capacity, and are only meant to "align" representations, rather than improving the model.
Comparison Representational Similarity Metries. Much of the current work studying represen- tations focuses on similarity metrics such as CKA. However, we argue that model stitching can be a better suited tool study representations conclusions about the behavior of neural representations compared to these metrics. We analyze the differences between model stitching and prior similarity metrics in Section 3.
various scenarios, and can give qualitatively different
Quantitative evidence for intuitions. Using model stitching, we are able to provide formal and quantitative evidence to the intuitions mentioned above. In particular we give evidence for the "all roads lead to rome" intuition by showing compatibility of networks with representation that are trained using (1) different initializations, (2) different subsets of the dataset, (3) different tasks (e.g., self-supervised or coarse labels). See Figure 2A-B for results. We also show that network with different random initialization enjoy property which we call stitching connectivity, wherein almost all minima reachable via SGD can be "stitched" to each other with minimal loss of accuracy. We also give evidence for the "more better" intuition by showing that we can improve the performance of a network A by plugging in a representation r that was trained with (1) more data, (2) larger width, or (3) more training epochs. See example with more samples in Figure 2C.
The results above are not surprising, in the sense that they confirm intuitions that practitioners might already have. However model stitching allows to obtain quantitative and formal measures of these in a way that is not achievable by prior representation measures.
Organization. We first define model stitching formally in Section 2. We compare it with prior work on representational similarity measures in Section 3. Then, we formally define stitching connectivity and provide experimental evidence for it in Section 4. Finally, in Section 5 and 6, we provide quantitative evidence for the "all roads lead to Rome" and "more is better" intuitions respectively.
Related works. As mentioned above, stitching was introduced by Lenc and Vedaldi [2015] who studied equivalence of representations. They showed that certain early layers in a network trained on the Places dataset [Zhou et al., 2014] are compatible with AlexNet (see Table 4 in Lenc and Vedaldi [2015]). After completing this work, we were made aware of concurrent work Csiszárík et al. [2021] that also proposes model stitching compare neural representations. The results from this
3

---

work complement ours by studying the effect of changing the stitching layer in various ways (for instance, imposing a sparsity penalty on the stitching layer), and further clarifying the relationship of stitching with other representational similarity measures. In contrast, our work demonstrates stitching compatibility under different scenarios, like the comparison between supervised and self-supervised methods and experiments that show "more is better".
Our work on stitching connectivity is related the work on mode connectivity [Garipov et al., 2018, Freeman and Bruna, 2017, Draxler et al., 2018) 18.. These works show that the local minima found by SGD are often connected through low-loss paths. These paths are generally non-linear, though it was shown that there are linear paths between these minima if they are identically initialized but then use different SGD noise (order of samples) after a certain point in training [Frankle et al., 2020]. Stitching connectivity is complementary to mode connectivity. Stitching connectivity corresponds to a discrete path (with as many steps as layers) but one where the intermediate steps are interpretable. We also show stitching connectivity of networks that are trained on different tasks. Finally, most of the prior work on concrete metrics for relating representations was in the context of representation similarity measures. We describe this work and compare it to ours in Section 3.
2 Model Stitching
Let A be a neural network of some architecture A, and let X' - R2 be a candidate "representation" function. We consider a family S of stitching layers which are simple (e.g. linear 1 x convolutional layers for a convolutional network A) functions mapping R2 to Re where de is the width of A's E-th layer. Given some loss function L (e.g.. CIFAR-10 or ImageNet test accuracy) we define
Ar(r;A) inf ((A)) #ES
where Ast denotes the function mapping the activations of A's êth layer to the final output, and denotes function composition. That is, (r; A) is the smallest loss obtained by stitching into all but the first l layers of A using a stitching layer from S. We define the stitching penalty of the representation r with respect to A (as well and the loss L) as Le(r; A) [(A). If the penalty is non-negative, then we say that the representation r is at least as good as the first E layers of A.
Consider the simple case of fulnecconnected network and stitching family of linear functions. In this case, model stitching tells us there is a way to linearly transform the representation r into that of the first E layers of A, but only in the subspace that is relevant to the achieving low loss. In practice, we approximate the infimum in Equation 1 by using gradient methods- concretely, by optimizing the stitching-laye using the train set of the task L. The stitching penality itself is then estimated on the test set.
Choosing the stitching family S: The stitching family can be chosen flexibly depending on the desired invariances between representations, as long as is simple (see below). For instance, if we choose S to be all permutations orthogonal matrices, the stitching penalty will be invariant up to permutations or orthogonal transformations respectively. In this work, we mainly consider cases where B<i for B with architecture similar to A. We restrict the stitching family per architecture A such that the composed model Ayε o 8 o r lies in A. This way the stitched model consists of layers identical to the layers of either A or B, with the exception of just one layer. For example, for convolutional networks we consider a x 1 convolution and for transformers we consider a token-wiss linear function between transformer blocks. We perform an ablation with kernels of different sizes in Appendix B.I.
Stitching is not learning: One concern that may arise is that if the stitching family S is sufficiently powerful, it can learn to transform any representation into any other representation. This would defeat the aim of faithfully studying the original representations. To avoid this, the family of stitchers should be chosen to be simple (c.g., linear). Nevertheless, to verify that our experiments are not in such a regime, we stitch an untrained, randomly initialized network with fully trained network. Figure 2A- shows that the penalty of such a stitched network is high, specially for high I. CKA has the same trend for the same networks (See Appendix B.2)
Model stitching as a "happy middle". Model stitching can be considered as a compromise between the following extremes: (1) Direct plugging in: the most naive interpretation of "plugging in" r into A would be to use no stitching at all. However, even in cases where networks are identical up to a permutation of neurons, plugging in into A would not work. (2) Full fine tuning: the other extreme

---

interpretation is to perform full fine tuning. That is, start with the initialized network A>e r and optimize over all choices of A. The problem with this approach that there are so many degrees of freedom in the choices for A> that the resulting network could achieve strong performance regardless of the quality of r. For example, in B.3 we show that fine tuning can fail to distinguish between trained network and a random network. (3) Linear probe: popular way to define quality of representations is to use linear probes [Alain and Bengio, 2016]. However, linear probes are not as well suited for studying the representation of early layers, which have low linear separability. In particular, the linear probe accuracy of an early layer of network A would generally always be much worse than later layer of network B, even if A was of "higher quality" (c.g., trained with more data). Linear probes also don't 't have the operational interpretation of compatibility.
By choosing a trainable but law-cipy layer, model stitching "threads the needle" between simply plugging in, and full fine tuning, and unlike linear probes, enables the study of early layers using powerful nonlinear decoders.
Experimental setup. Unless specified otherwise, the CIFAR-IO experiments are conducted on the ResNet-18 18 architecture (with first layer width 64) and the ImageNet experiments are conducted on the ResNet-50 architecture [He al., 2015]. The ResNets are trained with the standard hyperparameters (See Appendix A. for training parameters of all base models). The stitching layer in the the convolutional networks consist of of channels in and output features equal to the output channels of (<( We add a BatchNorm (BN) layer before and after this convolutional layer. Note that the BÑ layer does not change the representation capacity of the stitching layer and only aides with optimization. We perform stitching only between ResNet blocks (and not inside a block), but note that it is possible to stitch within the block as well. We use the Adam optimizer with the cosine learning rate decay and an initial learning rate of 0.001. Full experimental details for each experiment are described in Appendix A.
1 convolutional layer with input features equal to the number
Stitching vs. representational similarity
Much prior work studying representations focused on representation similarity measures. These have been studied in both the neuroscience and machine learning communities [Kriegeskorte et al., 2008, Kornblith et al., 2019]. Examples of such measures include canonical correlation analysis (CCA) [Hardoon et al., 2004] and its singular-vector and projection-weighet variants such as SVCCA and PWCCA (Raghu et al., 2017, Morcos et al., 2018). Recently Kornblith, Norouzi, Lee, and Hinton [2019] proposed centered kernel alignment ( (CKA) that addressed several issues with CCA.
CKA was further explored by Nguyen et al. [2021].
For two representations functions Φ X Rd and X + Hdd the linear CKA is defined [[Cov(ott)
as CKA(Ø, a) = [Cov(o(z)]]e [Cov(o ( where all covariances are with respect to the test distribution on inputs x ~ D [Krmblithh et al., 2019].
Sritching Connectivity Different Initialization
"All roads" Self-Supervision
"More is better"
Method
More Data/ / Time / Width
CKA Stitching
Varies (can be 0) Close (up to 3% error)
Varies (0.35 0.9)
Far (can be 0 for data, 0.7 for width) Better
Close (up to 5% error)
Table I: Qualitative results of our experiments, comparing CKA to stitching. CKA R 1 means representations are close according to CKA. Error z 0% means representations close according to stitching. "Varies" means no consistent conclusion across different architectures and tasks.
Table 1 contains qualitative summary of our results, comparing stitching with CKA. While in some cases the results of stitching and CKA agree, in several cases, stitching obtains results that align more closely with the intuitions that perf-oerg networks learn similar representations, and that more resources results in better versions of the same representations. In particular, in experiments where stitching indicates that a certain representation indicate that the two representations are far from each other. Moreover, in some experiments CKA indicates that representations are far where we intuitively believe that they should be close, e.g. when networks only differ by two random initializations, when one is trained with a supervised and
better than another, CKA by its design can only
5

---

another with self-supprvised task.' In contrast, in these settings, model-stitching reveals that the two models have nearly equivalent representations, in the sense that they can be stitched each other with low penalty. The precise experimental results appear in Sections 4-6 and Appendix B.2.
Compared to representatioonssiilaayy measures, model stitching has several advantages:
Ignoring spurious features: Measures such as CCA and CKA ultimately boil down to distances between the feature vectors, but they do not distinguish between features that are learned and relevant for downstream tasks, and spurious features, that may be completely useless or even random. For example, suppose augment representation é X - R by concatenating 1000 "useless" coordinates, with random gaussian features, to form a new representation d' : X + Rd R 1000 This would reduce the CKA, but representation o' is not different from 6 in meaningful way. Model- stitching resolves this, since we can stitch ø' in place of & by simply throwing away the useless coordinates. In general, model-stitching focuses only on aspects of the representation which are relevant for the downstream task, as opposed to aspects which are spurious or irrelevant. The price we pay for this is that, unlike measures such as CKA, model stitching depends on the downstream task. However, our results indicate that neural networks tend to learn similar representations for a variety of natural tasks.
Asymmetry: Our intuition is that some representations are better than others. For example, we believe that with more data, neural networks leam better representations. However, by design, such comparisons cannot be demonstrated by representation similarity measures that only measure the distance between two representations. In contrast, are able demonstrate that "more is better" using saaching-hasedd measures. Concretely, CKA and other similarity measures are symmetric, while stitching is not: it may be the case that representation can be stitched in place of representation o, but not vice-versa. Also, stitching a representation & can (and sometimes does) improve performance.
Interpretable units: Measures such as CCA/CKA give number between 0 and for the distance between representations, but it is hard to interpret what the difference, for example, between a CKA value of 0.9 and value of 0.8. In contrast, if the loss function L has meaningful units, then the stitching penalty inherits those, and (for example) a representation r having a penalty of 3% in CIFAR- 10 accuracy has an operational meaning: if you replace the first layers of the network with r, the decrease in accuracy is at most 3%.
Invariance: A representation similarity measure should be invariant to operations that do not modify the "quality" of the representation, but is not always clear what these operations are. For example, CCA and CKA are invariant under orthogonal linear transformations, and some variants of CCA are also invariant under general invertible linear transformations (see Table 1 in Kornblith et al. [2019]). However, it is unclear if these are the natural families. For example, randomly permuting the order of pixels (either at the input or in the latents) is an orthogonal transform, and thus does not affect the CKA. However, this completely destroys the spatial structure of the input, and intuitively should affect the "representation quality." On the other hand, certain non-orthogonal transforms may still preserve representation quality- for example, the non-invertible transformation of projecting out spurious coordinates. Thus, orthogonal transforms may not be the right invariance class to consider representations. Using stitching we can explicitly ensure invariance under any given family of transformations by adding it to the stitching layer. Concretely, our choice of using a 1 x 1 convolutional stitcher yields much weaker invariance than general orthogonal transforms- and thus, model stitching can predict that shuffling pixels leads to a "worse" representation.
4 Stitching Connectivity
We first focus special case of model-stitching, wherein stitch two identically distributed networks to each other. That is, we train two networks of same architecture, on the same data distribution, but with independent random seeds and independent train samples. This question has been studied before with varied conclusions [Li et al., 2016, Wang et al., 2018]. We find that empirically, two such networks can be stitched to each other at all layers, with close to 0 penalty. This is new empirical property of SGD trained networks, which we term "stitching connectivity."
For the former case, this is highlighted in Figure 6 of Nguyen et al. [2021], where it is stated that "Represen- tations within block structure differ across initializations."

---

Formally, let A, B be two trained networks of identical architectures with L layers, for a task with objective function L. For all i {0, I...,, L}, define S; be the stitched model where we replace the first i layers of A with those same layers in B (and optimize the stitching layer as usual). That is, S. == arg s= ((S). Observe that So = A and St. B, so the sequence of models (S, Si,..., }r.} gives a kind of "path" between models A and B. Further, due to our family of stitching layer and network architecture x conv stitchers and conv-nets), the stitching layer can be folded into the adjacent model. Thus, all models S; have identical architecture as A and B. We say A and B are sattching-connecter if all the intermediate models S, have low test loss.
Definition 1 (Stitching Connectivity). Let A, B be two networks of identical architectures. We say A and B are That is, if all stitched models S₁, defined as above, have test loss comparable to A.
if they can be stitched to each other at all layers, with low penalty.
Stitching connectivity is not a trivial property: two networks with identical architectures, but very different internal representations, would fail be stitching connected. Our main claim is that for fixed data distribution, almost all minima reached by SGD are
to each other.
Conjecture 2 (Stitching Connectivity of SGD, informal). Let A1, A2 be two independent and cenntically-trainnee networks. That is, networks of the same architecture, trained by SGD with independent random seeds and independent train sets from the same distribution. Then, for natural architectures and datradistributtonss the trained models A and A2 are stiching-coonectee.
This conjecture states a structural property of models that are likely to be output by SGD. If we run the identical training procedure twice, we will almost certainly not produce models with identical parameters. However, these two different parameter settings yield essentially equivalent internal representations- this is what it means to be stitching-connecte
Discussion. Stitching connectivity is similar in spirit to mode connectivity [Garipov et al., 2018, Freeman and Bruna, 2017, Draxler et al., 20188, (018].7m in that they are both structural properties of the set of typical SGD minimas. Mode connectivity states that typical minima are connected by a low-test-loss path in parameter space. Stitching connectivity does not technically define a path in parameter space- rather, it defines sequence Su. Si,... S2 of low-loss models connecting two endpoint models. For each model in this sequence, all but one layer is identical to one of the endpoint models. Thus, we can informally think of the stitching sequence as a different way of "interpolating" between two models.
The stitching connectivity of SGD especially interesting for overparameterized models, since it sheds light on the implicil bias of SGD. In this case, there are exponentially-many global minima of the train loss, even modulo permutation simiratiionnssmmtty..... Apriori, it could be the case that each of these minima compute the classification decision in different ways (c.g. by memorizing a particular train set). However, empirically we find that SGD is "biased" towards minima with essentially the same internal representations- in that typical minima can be stitched to each other.
Experiments. Figure 2A demonstrates the fettching-connectiine ment. We train a variety of model architectures on CIFAR-10, 10, with two different randomly initialized models per architecture. We consider ResNet-18, 18, two variants of ResNet-18 18 with 0.5x and 2x width, a significantly deeper ResNet- 164, a feed-forward convolutional network Myrtle-CNN [Page, 2018] and Vision Transformer (Dosovitskiy al., 2020] pretrained CIFAR-5m [Nakkiran et al., 2021]. We stitch the two randomly initialized models at various intermediate layers, forming the stitching sequence Sp. ... St. Figure 2A plots the test errors of these intermediate stitched models S, with the first and last point showing the errors of the base models. For all layers i, the test error of the stitched model S, is close to the error of the base models. A similar result holds for networks trained on disjoint train sets. Full experimental details are provided in Appendix A.
of SGD via the following experi-
5 All Roads Lead to Rome
In this section, we quantify the intuition that "all roads lead to Rome" in representation learning: many diverse choices of train method, label quality, and objective function all lead to similar representations in early layers. However, we find that such training details can affect the representation at later layers.
Comparing self- supervied and supervised methods. If we train models with the same architecture and train set, but alter the training method significantly, what do we expect from the representations? To explore this, we compare the representations of two very different training methods - standard
7

---

end-to-end supervised training (E2E) versus self- self-supervised + simple classifiers (SSS), that first learn a representation from unlabeled data and then train simple linear classifier on this representation.
SSS algorithms like SimCLR [Chen et al., 2020b], SWAV [Caron et al., 2021a] and DINO [Caron et al., 2021b] etc have recently emerged prominent paradigm for training neural networks that achieve comparable accuracy to E2E networks. We stitch the representations of these SSS algorithms to an E2E supervised network (all with a ResNet-50 backbone) trained on ImageNet. While all of these methods achieve similar test accuracy on a ResNet-50 backbone of 75% + 1 (except SimCLR at 68.8%) [Goyal et al., 2021], they are trained very differently. Figure 2B shows that the SSS trained networks are stitching connected at all layers to the E2E network. This suggests that while the advances in SSL have been significant for learning features without labels, the features themselves are similar in both cases. This is in agreement with prior work which shows that SSS and E2E networks have similar texture-shape bias and make similar errors [Geirhos et al., 2020J. Since certain SSS algorithms have been proven to have small generalization gap [Bansal et al., 2021], the similarity of representations between SSS and E2E algorithms may yield some clues into the generalization mystery of E2E algorithms, A similar experiment for SimCLR with ResNet-18 trained on CIFAR-10 is shown in Appendix B.2, along with the CKA for the same networks. We find that the CKA varies between 0.35- 0.9 for different layers, while stitching gets maximum stitching penalty of 3%.
Changing the label distribution p (y)x:): How much does the representation quality depend on label quality? To explore this, we compare the representations of networks with the same imput distribution p(z), but different label distributions p(y [z). We take the CIFAR- 10 distribution on inputs p p(z). and consider several "less informative" label distributions: (1) Coarse labels: We super-class CIFAR-10 10 classes into binary task, of Objects (Ship, Truck, etc.) vs. Animals (Cat, Dog, etc.) (2) Label noise: We set p {0.1, 0.5, 1.} fraction of labels to random labels. We then stitch these "weak" networks to a standard CIFAR-10 network at varying layers, and measure the stitching penalty incurred in Figure 3A. We find that even with poor label quality, the first half of layers in the "weak" model are "as good as" layers in the standard model (with the exception of the weak network trained on 100% noisy labels). These experiments align with the results of Nakkiran and Bansal [2020], which show that neural networks can be sensitive to aspects of the input distribution that are not explicitly encoded in the labels. Full experimental details appear in Appendix A.
These results are in agreement with prior work on vision [Olah et al., 2017], which suggests that the first few layers of neural network learn general purpose features (such as curve detectors) that are likely to be useful large variety of tasks. Formalizing the set of pre-training tasks for which such similar representations are learnt is an important direction for future work to understand pretraining.
6
More is Better
Now, we use stitching to quantify the intuition that "more is better" for representations. That is, larger sample size, model size, or train time lead progressively better representations of the same type.
Number of samples: When scaled appropriately, neural network performance improves predictably with the number of training samples (e.g. [Kaplan et al., 2020]). But how does this improvement manifest in their representations? We investigate this by stitching the lower parts of models trained (5K, 10K, )5 samples of CIFAR-10 to the upper layers of a model trained with 10K samples.
First, as Figure 2C shows, we observe that the models are all stitching compatible- the better representation trained with 25K samples can simply be "plugged in" to the model trained with fewer samples, and this improves the performance of the stitched network relative to the 10K network. Note that there is no theoretical reason to expect this compatibility, better networks could have learnt fundamentally different features from their weaker counterparts. For example, a network trained on few samples could have learnt only "simple features" (presence of sky in the image / presence of horizontal lines), while one trained on many samples may learn only "complex features" (presence of an blue eye/ / presence of furry car), which are not decodable by the weaker network.
Secondly, we find that some layers are more data hungry than others. When stitched at the first few layers, all the models have similar performance, but the performance degrades rapidly with fewer samples in the mid-layers of the network. This suggests that each layer of a neural network has its
We use this approach rather than the more standard SSL approach of self- supervieed training with fine tuning to ensure that all layers except the last were learned without the use of labels.
8

---

80-
tysste madel
12-
0.125
10-
S0uchigg hauult (5)
195177
80 Specha
8-
motel
Label
10-
100% Label Noise
20-
10-
bp model Eiroe
Top model Emor- 0.50%
A
0.0
0.6 Praction layers from bottom model
0.2
0.6
B
0.2 0.4
0.0
C
0.0
0.2
0.4
0.6
0.8
Fraction of ayers from bottom model
Fraction of layerss from bottom model
Figure 3: (A) Changing the label distribution: Representations trained on CIFAR-10 10 with the Object vs. Animals task or with 110% 50%, 100%) label noise and stitched to a network trained on original CIFAR-10 labels. Early layers learn similar representations is "less informative" than training with all labels (B) Increasing training time: Representations different epochs during training are stitching compatible and early layer converge faster (B) Increasing width: Better representations from a wider network can be stitched with a thinner network to improve performance. All experiments were performed with ResNet-18 on CIFAR-10.
when the label distribution
own sample complexity. As a corollary of this finding, we predict that we can train some of the layers with few samples, freeze them and train the rest of the model with larger number of samples. Indeed, we find that we can recover most of the accuracy of the network by training the first three, and the last three layers of this model (about half of the layers) with just 5K samples, freezing them and training the rest of the model with all the samples to obtain network within 2% accuracy of the original network. See Appendix B.4 for details of the experiment and training plots. This similar to the "freeze- "freeze-training" training" experiments suggested by Raghu et al. [2017].
We observe similar results when stitch networks trained for different number of epochs. We train a ResNet-18 18 trained on CIFAR-10 and stitch the representations from the model at (40,8) 80, 160} epochs to the model at the 80-th epoch. As training time increases, Figure 3B shows that the representations improve in a manner that compatible with the earlier training times. Note that this is nontrivial statement about neural network training dynamics: It could have been the case that, once a network is trained for very long, its representations move "far away" from its representations near initialization and thus, stitching would fail. However, representations at the end of training remain compatible with those in the early stage of training. We also observe that earlier layers converge faster with time, as was shown by Raghu et al. [2017].
Training time:
find that the
Width: Similarly, we train models with the same architecture but varying width multipliers x0.25x, 1x, 2x} } (Figure 3C). We find that "better" models with higher width models can be stitched to those with lower width and improve performance, but not vice versa. The CKA between represen- tations with 0.25x and 2 width multiplier is in the range 0.7 0.9 (See Appendix B.2)
Taken together, these results suggest that neural networks obey a certain kind of modularity better layers can be plugged in without needing to re-train the whole network from scratch.
7 Conclusion and Future Work
As our work demonstrates, model stitching can be a very useful tool to compare representations in interpretable units. Model stitching does have its limitations: it requires training network, making it more expensive in computation than other measures such as CKA. Additionally, stitching representations from two different architectures can be tricky and requires careful choice of the stitching family. While we restricted ourselves to stitching the first / layers of a network, stitching can be used to also plug in intermediate layers or parts of layers, and in general to "assemble" new model from a collection pre-trained components.
There are various avenues for future research. All of our results are in the vision domain, but it would be interesting to compare representations in natural language processing, since language tasks tend to have more variety than vision tasks. It would also be interesting to study the representations of adversarially trained networks diagnose why they lose performance comparison to standard training. In general, hope that model stitching will become part of the standard diagnostic

---

repertoire of the deep learning community. Societal impacts: This paper makes methodological and foundational contributions that do not have direct impact on society. Model stitching can potentially be used to understand and develop better representation learning mechanisms. While this could indirectly lead to future applications, it premature to predict their positive or negative impacts.
Acknowledgements
YB is supported by IBM Global University awards program and NSF Awards IIS 1409097. PN is supported in part by a Google PhD Fellowship, the Simons Investigator Awards of Boaz Barak and Madhu Sudan, and NSF Awards under grants CCF 1565264, CCF 1715187. BB is supported by NSF award CCF 1565264, a Simons Investigator Fellowship and DARPA grant W91 INF2010021. We thank MIT-IBM Watson Al Lab and John Cohn for providing access and support for the Satori compute cluster.
References
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644 2016.
Yamini Bansal, Gal Kaplun, and Boaz Barak. For self-supervised learning, rationality implies
generalization, provably. In International Conference on Learning Representations, 2021. URL htttps://openrevier wwwpeeeiwwwwwweeeeeeeeeeeeeeeeee /tt.:/rrrmmmddSSmgg33333333
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments, 2021a.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers, 202
Mark Chen, Alec Radford, Rewon Child, Jeff Wu, Heewoo Jun, Prafulla Dhariwal, David Luan, and Ilya Sutskever. Generative pretraining from pixels. 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, 2020b.
Adrián Csiszárik, Péter Kōrösi-Szabó, Ákos K. Matszangosz, Gergely Papp, and Dániel Varga. Similarity and matching of neural network representations. Advances in Neural Information Processing Systems, 35, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2020.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International conference on machine learning, pages 1309-1318. PMLR, 2018.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pages 3259-3266 PMLR, 2020.
C. Daniel Freeman and Joan Bruna. Topology and geometry of half-rectified network optimization, 2017.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew Gordon Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns, 2018.
Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. On the surprising similarities between supervised and self-supervised models. CoRR, abs/2010.08377, 2020. URL https: www.xii.oog/bbbb..
10

---

Gabriel Goh, Nick Cammarata 9, Chelsea Voss 4, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 2021. doi: 10.2391 5/distill.00030 000000.
Priya Goyal, Quentin Duval, Jeremy Reizenstein, Matthew Leavitt, Min Xu. Benjamin Lefaudeux, Mannat Singh, Vinicius Reis, Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Ishan Misra. 2021.
Vissl. https:/ htte:://gthubb
David R Hardoon, Sandor Szedmak, and John Shawe-Taylor. Canonical correlation analysis: An overview with application to learning methods. Neural computation, 16(12):263--264, 2004.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
Don Komarechka. 2021.
Images on https://www.macee
ca/ h//utuure/arts//
american canadiann canadian--noufllee--- haa.aaaaamoooaaak--hhhoorrrr///a
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey E. Hinton. Similarity of neural network representations revisited. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 3519 3529. PMLR, 2019. URL proceedince. mlr war..press/vie grres/v97/kornbllr /kornblitht9a. html.
Nikolaus Kriegeskorte, Maricke Mur, and Peter Bandettini. Representational similarity analysis connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2:4, 2008. doi: 10.3389/neuro.06 00..ro.00000.00000000000 URL https wrwttirrnoniiiiiinnnn org/ article/10 33811ceee11.31//////// 13389/neuro 06. 006.0.04..0 2008.
ISSN 1662-5137.
Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equivariance and equivalence. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 991-999, 2015.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 6391-6401, 2018.
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hoperoft. Convergent learning: Do different neural networks learn the same representations?, 2016.
Shengehao Liu, Dimitris Papailiopoulos, and Dimitris Achlioptas. Bad global minima exist and sgd can reach them. Advances in Neural Information Processing Systems, 33, 2020.
Ari S Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. In NeurlPS, 2018
Preetum Nakkiran and Yamini Bansal. Distributional generalization: new kind of generalization. arXiv preprint arXiv:2009.0800 .8002, 2020.
Preetum Nakkiran, Behnam Neyshabur, and Hanie Sedghi. The deep bootstrap framework: Good online learners good offline generalizers, 2021.
Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. ICLR, 2021.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. doi: 10.239 10.23915/distill //0iitill.00007.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. overview of early vision in inceptionvl. Distill, 2020. doi: 10.2391
11

---

David Page. How to train your resnet.
https://myrtle.at ai/
Now-to-traee how-to-train- peaain-oou-resneeee 4- architecture/ 2018.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. SVCCA: sin- gular vector canonical correlation analysis for deep learning dynamics and interpretability. In Advances Neural Information Processing Systems 30: Anmual Conference on Neu- ral Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA. USA, puges 6076-6085, 2017. URL https: /ttp///eeooce....e//ee neurips...... 466777665a11681006677334666666. cccbrbtraatctt html.
Abstract
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams, Learning internal representations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
Liwei Wang, Lunjia Hu, Jiayuan Gu, Yue Wu, Zhiqiang Hu, Kun He, and John Hoperoft. Towards understanding learning representations: To what extent do different neural networks learn the same representation, 2018.
Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learning deep features for scene recognition using places database, In Neural Information Processing Systems, 2014.
12


Reviews:
[{"review_id": "a6FnZ3-Orki", "reviewer": "Reviewer_myvx", "summary": "The paper use model stitching as a way to examine the hidden states of neural network models. The proposed method stitches the top and bottom of two similar models together with a linear transformation (if you don't take BatchNorm into account). Experiment results show that the hidden states, learned from different initializations, different subsets of the training set and even different losses can be translated from one to another. Such that the top of one model can use the hidden state given by another model to predict the same label as the original top model. Based on this observation the paper proposed stitching connectivity, a measurement of similarity between semantic space of hidden states.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall the paper is interesting and convincing. It studies a long-standing question of deep learning -- whether the hidden states of different models are modeling the same features.  \nThe idea of model stitching is already proposed in previous work (Lenc & Vedaldi 2015). But the paper proposes to use the stitching penalty as a quantitative metric to study the question.\nThe experiments are comprehensive. The author proposed several experiment settings to show the advantages of the \"stitching penalty\" over other metrics. Experiment results also show that good networks of the same architecture, but trained in very different ways can be stitched to each other without a drop in performance. \nThe conclusion should be interesting to a wide spectrum of researchers, for example, whether pre-trained language models learned the same set features. \nThe paper is well-written. Although the paper is not organized in a regular way, presenting detailed results in the introduction actually makes the paper easier to read.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. Like most theoretical papers, the conclusion of this paper is based on two assumptions:\na) linear transformation with BatchNorm is not a strong translation mechanism that can map one type of representation to another;\nb) the inputs (hidden states), that provide similar outputs (labels), should share a similar representation space.\nAlthough the two assumptions seem intuitively correct, more theoretical or empirical evidence will further strengthen the paper.\n\n2. It would also be interesting to examine what information or features are preserved through the stitch linear. Since the author argued that the stitching layer can ignore spurious features.\n\n3. Line 256-259: All experiments are done with the ResNet model and Adam optimizer, but the author generalized their conjecture to SGD and neural network models. It seems overclaimed.\n\n4. The related works section is not comprehensive, it didn't include other works that study similar questions.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "GEJllZ4ODYI", "reviewer": "Reviewer_Anx8", "summary": "The work proposes to utilize model stitching as a tool to compare the quality of learned neural representations. Authors show that model stitching based comparison can avoid many problems often observed in other representation comparison methods, such as CCA/CKA. Using this tool, under image classification setting, authors empirically show that (1) networks with different random initializations would converge to \"similar\" representations, (2) networks trained via different methods also have \"similar\" representations given these networks have close performances, and (3) networks trained with more data consistently have better quality. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem of comparing learned neural representations is definitely an important one. And the solution proposed in this work is well motivated and can provide more accurate and richer information of the representation compared to other methods. The conclusions made in the work are also very intriguing.\n\nThe main limitation of the current study is the stitching is only performed on a pair of models with exactly the same architecture. If the tool could only be used in this way, its value would be quite limited and the corresponding conclusions would become less interesting.\n\nI was wondering why authors didn't try to stitch models of different architectures. \n- One simplest case is to stitch two models with similar computation cost but different hyper-parameters from the same architecture family, for example a ResNet-18 and a ResNet-50 with 0.5x width. Effectively, this would provide meaningful information about the depth/width trade-off. Also, the experiment should be easy to conduct.\n- More general case would be stitching models from different architecture families. This test is also important in practice, as people often hope to compare representations from different architecture families. If model stitching does not provide meaningful signal here, this would significantly restrict its value.\n\nAnother limitation also acknowledged by the authors is that all experiments are done in image classification. Hence, it remains unclear whether the method is general enough for other problems (say language pretraining where lots of pretrained models have been open sourced).\n\nA minor issue is the stitching penalty is originally defined as the loss gap (Line 120). But in Figure 1, percentages are used to present the stitching penalty, which is inconsistent. \n\nIn summary, I feel the work is targeting an important problem and has proposed a well motivated method. However, the empirical study is relatively weak, making it difficult to judge the generalness of the proposed method and hence its practical value.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "For limitation, see the comments above. I don't see any concern w.r.t. negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "DuPrjqEsD6a", "reviewer": "Reviewer_tKuq", "summary": "Understanding how the representations of neural networks, trained in diverse settings, compare and can be combined is an important problem in representation learning. This papers addresses this problem by studying model stitching (Lenc & Vedaldi 2015) in the following experimental setting: connect the bottom-layers of A to the top-layers of B with an inexpensive trainable layer, where A and B are trained neural networks. The contributions are:\n\n1) Empirical confirmation of inutitive statements: such as: \n\n* representations trained with more data and training time or with larger width can be stitched to representations trained with less resources;\n\n* representations trained with different objectives (supervised vs. self-supervised) can be stitched together with minimal penalty;\n\n* early layers of representations are similar even when the labels used for training are different.\n\n2) Observation of a new structural property of SGD, defined as \"stitching connectivity\" by the authors: typical minima reched by SGD can be stiched together without changing accuracy much.\n\n\n\n\n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Originality\n\nThe idea to revisit model stitching is timely. While intuitive, the experiments are cleverly designed, and demonstrate the benefits from model stitching.\n\n# Quality\n\nThe study is methodical and useful. While reading the paper, most of my questions were eventually answered in later parts of the text. \n\n# Clarity\n\nThe paper is overall clear, well-organized and engaging. Some suggestions for improvement: \n\n* Line 103: Garipov et al. is missing a year.\n\n* Line 122: simple of case <- simple case\n\n* Line 133 is ambiguous: “lies in \\mathcal{A}” is not very formal, as you haven’t defined what \\mathcal{A} is formally. Line 113*  doesn’t help. Why is the composition part of \\mathcal{A}. You answer my question in lines 246-248, but I think you should make this comment earlier. \n\n* Table 1: What does \\textbf{Better} mean here? “Far” for CKA means representation similarity, but “Better” means that stitching improves the method. These comparisons are different. You address my question in line 189, but I think you should note it in the Table’s caption.\n\n* Line 212-214: Have you studied this possibility? It might be interesting.\n\n* Line 249: “low test loss” is not a formal definition. Does that affect your discussion?\n\n* Line 342-351: this is an interesting experiment, but it may not be very realistic. If the labels are available, why not use them to update the first and last layers too? Apart from reducing computation for training, is there any situation where freezing the first and last layers would be better?\n\n* Lines 78-79: “much better suited tool” <- I would suggest to rephrase this, because studying representations should be more specified. Depending on what one wants to study, different tools might be preferred. As the conclusion section of [Kornblith et al., 2019] clarifies, measuring similarity between representations is an “ill-defined problem.”\n\n* What is the legend of the colors in Figure 4 (in SM). It's not labeled currently.\n\n* Figure 6 (in SM): in the caption, \"differ more than two\" seems to be missing a word. Also, the captions talks about (a) and (b), but we do not see it in the Figure. SimCLR and E2E do not seem \"identical.\" \n\n\n# Significance\n\nThe convincing empirical results about model stitching would likely aid researchers and practitioners. The former can study interesting follow ups from this study, such as the \"sample complexity of individual layer\" and the later can improve applications, such as stitching two high-performing, but complimentary, neural networks.\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the big thing that the paper should discuss is why the stitching behavior happens for natural datasets. I imagine, given a fixed neural network architecture, one could design a dataset, where stitching connectivity would fail. In fact, it could be possible to optimize for such a dataset with meta learning. Any thoughts about that?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "AswDFanvjsu", "reviewer": "Reviewer_CLM9", "summary": "The authors studied model stitching as a methodology to examine the internal representations of neural networks. Particularly, they use model stitching to verify several statements such as “good networks learn similar representations” and “representations learned with (1) more data, (2) bigger width, or (3) more training time can be “plugged in” to weaker models to improve performance\". In addition, They showed that SGD has a structural property akin to mode-connectivity: typical minima reached by layers trained via SGD can all be stitched to each other with minimal change in accuracy, termed as  “stitching connectivity”.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The work is only marginally innovative since the idea of model stitching was introduced a long time ago by Lenc and Vedaldi (2015) and the current work only extends the method to make it applicable in new contexts. It’s nice that the work shows that model stitching is more powerful than originally conceived, and it has several benefits over representation-similarity measures, such as ignoring spurious features, asymmetry, interpretable units, and invariance. However, the innovation is still incremental.\n\n The submission is relatively sound. The authors verify several intuitive statements such as “good networks learn similar representations” and “more is better” with the model stitching approach. I do have a couple of concerns. In section 4 Stitching connectivity, the author mentioned “We find that empirically, two such networks can be stitched to each other at all layers, with close to 0 penalty. This is a new empirical property of SGD trained networks, which we term “stitching connectivity.” However, the conclusion is based on experiments with a limited set of computer vision models that are primarily convolutional networks on a single dataset CIFAR-10. I doubt that any general argument about the empirical property of all SGD trained networks,  i.e., “stitching connectivity.” could be derived from these limited experiments. \n\nMy second concern is about line 318: “We find that even with poor label quality, the first half of layers in the “weak” model are “as good as” layers in the standard model. Even when the model is given less informative labels, such as coarse labels (object vs. animal) the first few layers in the weak network learn representations as good as the “strong” networks\". I think this is very counter-intuitive and perhaps very experiment-specific. It’s likely due to the fact that the original CIFAR-10 dataset only has ten classes and reducing 10 classes to 2 classes does not make a big difference. I wonder if the authors have tested the model on a dataset having much more classes like ImageNet. If reducing the class numbers still leads to near-zero stitching penalties in the first few layers in the “weak” neural network, then the author’s argument that “the first few layers of a neural network learn general purpose features (such as curve detectors) ” (line 324) would be more convincing.\n\nMy third question is related to the training of the stitcher. In Appendix Line 565, the authors give a brief description of the procedure of training the stitcher. I’m curious to know the number of epochs for the stitcher training (if fixed)  or any strategies for early-stopping if used. This is important because the stopping criteria should be the same for training the stitcher and training the original model. Otherwise, the performances can not be fairly compared and the stitching penalty cannot be reliably obtained.\n\nThe submission is clearly written and well organized. If the authors can address the issues I mentioned above and make the arguments and conclusions in the paper more convincing, the results can be important in the sense that it shows model stitching can be a powerful tool to study representation learned by deep nets. Particularly, this method can be used to compare representation similarity, determine which representation is better and understand the impact of different representations on model performance via stitching penalty.  Future studies could use this approach to understand deep neural nets better. \n\n-----\n\nUpdate: I've read the authors' responses to my review as well as the ones to the other reviewers'. I think their responses addressed some of my questions. I appreciate the clarification on the training procedure. I still think that the authors need to make it clear that the property observed for the SGD trained networks in this paper are purely empirical and whether it generalizes beyond the compute vision models requires additional experiments in future research. Given the improvements and the limitations, I decided to raise my score to 7 but no further.  \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors note that there’s no obviously foreseeable negative societal impact of the work, which I largely agree with.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores a previously described technique called "model stitching", in which two trained models can be appended.  Through several empirical experiments the authors show that this technique provides insight into the representational spaces learned by the separate models. 

The reviewers thought this idea was interesting, and the paper was creatively written and clear.  The strongest critiques were about the thoroughness of experiments, stating that the authors did not explore the stitching of very diverse architectures or training on very different datasets.  However, I feel the analyses in this paper are extensive, and so I think it is reasonable to leave some of that further experimentation for future work. 
Decision: Accept (Poster)