Functional Alignment Can Mislead: Examining Model Stitching
Authors: Damian Smith
Harvey Mannering
Antonia Marcu
Venue: ICML 2025 spotlightposter (2025)
Keywords: stitching
model comparison
representation comparison
representation alignment
functional alignment
functional similarity
Primary area: deep_learning->other_representation_learning
TL;DR: Informational similarity must be accounted for when comparing models; model stitching does not do this.

Abstract: A common belief in the representational comparison literature is that if two representations can be functionally aligned, they must capture similar information. In this paper we focus on model stitching and show that models can be functionally aligned, but represent very different information. Firstly, we show that discriminative models with very different biases can be stitched together. We then show that models trained to solve entirely different tasks on different data modalities, and even clustered random noise, can be successfully stitched into MNIST or ImageNet-trained models. We end with a discussion of the wider impact of our results on the community's current beliefs. Overall, our paper draws attention to the need to correctly interpret the results of such functional similarity measures and highlights the need for approaches that capture informational similarity.

Functional Alignment Can Mislead:
Examining Model Stitching
Damian Smith Harvey Mannering'
Antonia Marcu
Abstract
Do networks that learn solve all these tasks really represent the same information?
A common belief in the representational compar- ison literature is that if two representations can be functionally aligned, they must capture simi- lar information. In this paper we focus on model stitching and show that models can be functionally aligned, but represent very different information. Firstly, we show that discriminative models with very different biases can be stitched together. We then show that models trained to solve entirely dif- ferent tasks on different data modalities, and even representations in the form of clustered random noise, can be successfully stitched into MNIST or ImageNet-trainn -eaiieeee- models. We proceed by show- ing that alignments can also be found in the case of autocncoders where the encoder and decoder are trained on different tasks. We end with dis- cussion of the wider impact of our results on the community's ' current beliefs. Overall, our paper draws attention to the need to correctly interpret the results of such functional similarity measures and highlights the need for approaches that cap- ture informational similarity.
Birdsong
Clustered-
Noise
Stylized ImageNet
ImageNet
Figure I. Examples of inputs from 4 different tasks we consider. We show that models trained on the first three tasks can stitched together with models trained on ImageNet despite depicting very different input-level information.
different, since they also use different input patterns for their predictions. Throughout the paper we refer to input-level patterns as "semantic" information.
We argue that without capturing semantic similarity, some of the existing model comparison measures can end up considering network that successfully distinguishes birds based their songs to be equivalent one that successfully distinguishes ImageNet images. In particular, we focus on model stitching but we argue in our discussion section that the same criticism might be extended to other measures and hypotheses as well.
1. Motivation
Measuring representational similarity is complex task and there is no consensus on how should be done (Sucholut- sky et al., 2023). More interestingly, there seems to be consensus on when two representations can be considered similar. In this paper we do not consider two representations to be similar if they do not capture the same information. For example, one model learns to classify birds by their song and another learns to classify them by their visual ap- pearance, we consider those models to be different. Equally, if two models learn visually classify birds but one of them uses the birds' shape make predictions, while the other uses plumage, we would consider those models to be
As an alternative to established model comparison methods such CKA (Kombllth et al., 2019). functional model com- parison is gaining increasing attention (Bansal et al., 2021; Hernandez et al., 2022; Klabunde et al., 2025; Csiszárik et al., 2021). In this paper we focus on the functional per- spective as an emerging direction and will therefore not cover CKA, SVCCA (Raghu et al., 2017), or similar meth- ods. Broadly speaking, the functional perspective argues that two intermediate layers should be considered similar if they lead to matching outputs. While this perspective seems useful, especially in classification setting, we argue that there is one fundamental problem with deep learning models which raises questions about its utility: shortcut learning (Geirhos et al., 2020).
'Vision. Learning, and Control (VLC) Research Group, Uni- versity of Southampton. Correspondence to: Antonia Mareu <a.marcu soton.ac.uk>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Shortcut learning is an umbrella term for learning decision

---

Functional Alignment Can Mislead: Examining Model Stitching
rules that rely on spurious correlations, causing the model to perform poorly outside of the setting it was trained on. Geirhos et al. (2020) argue that many challenges in deep learning boil down to shortcut learning. Moreover, the simplicity bias literature argues that SGID-trained models have a tendency to learn simpler rules (e.g. Valle-Perez et al., 2018; De Palma et al., 2019) (often shortcuts) over more complex rules that capture some of the true, salient, or intended information. Therefore, there is an increasing consensus that models are prone to picking up shortcuts in the data, whether we are aware of their existence or not. As result, it is important to be able to distinguish between models that learn different patterns because some could fail to generalise in the wild.
removing all shape information from the input data to be represented, leaving only colour information. We then stitch into model that was trained on shape information alone and that is unable to make correct predictions based on colour information. We achieve full stitching compatibility in this case as well. We then show that we can success- fully stitch clustered, random noise into the models we train. This, therefore, shows that we can easily construct cases where the models' stitching compatibility does not reflect their informational similarity.
In the second part of the paper we demonstrate that the problem is not restricted to simple settings by performing similar experiments with more complex, "real-world" data. For example, we show that we can stitch model trained on ImageNet (Russakovsky et al., 2015) to one trained to recognise bird songs.
In this paper we look at model comparison starting from the understanding that the models compare have a propen- sity for learning shortcuts. In particular, we focus on model stitching (Lenc& & Vedaldi, 2015), which is an increasingly popular method for comparing models' functional similarity. Informally, model stitching compares models A and B by "stitching" together the first part of model A (the sender) with the last part of model B (the receiver) using a linear transformation referred to as a "stitch". The stitch is trained such that it learns mapping between representations of models A and B. Model A is considered similar to model B if their stitched combination achieves an accuracy com- parable to that of model B. In other words, model stitching assesses how compatible two models are from a functional perspective by comparing how the function represented by their stitched combination behaves compared to the B func- tion when evaluated at the same input points. Functional compatibility is often taken to indicate that models cap- ture similar information (e.g. Lene& & Vedaldi, 2015; Bansal et al., 2021; Hernandez et al., 2022).
Our contributions are:
We argue that accounting for the extracted input pat- terns is of paramount importance when comparing rep- resentations, and show that model stitching does not achieve this;
We show that we can construct problems where stitch- ing cannot distinguish between models known to have learned different shortcuts (i.e. they use different infor- mation) (Section 4.1);
We show that representations that depict entirely differ- ent information or even clustered, random noise can be stitched into a trained model, raising further questions about the usefulness of stitching as a measure of model similarity (Section 4.2);
We extend our experiments to more complex, real- world, and benchmark datasets, considering distinct modalities and different tasks for both the discrimina- tive and the generative case (Section 5);
In this paper we design experiments which demonstrate that the stitching compatibility (or alignment) of models is inappropriate for evaluating the extent which two networks capture similar information. Importantly, model that learns spurious correlation can appear to be fully compatible with one that learns the intended information. Given the prevalence of shortcut learning, we argue that a meaningful model comparison should distinguish between models that capture different input patterns.
And, we end with discussion of the wider implica- tions of our results on functional alignment.
The code for our experiments is available at https: //gi thub. cwt.coo////////oe. DDH..mmiimi t st.tchiiiiin
We first show that models that learn to use different infor- mation can easily be stitched together. We start with toy problem where we can bias models towards learning to use either colour or shape information. We show that the mod- els are considered similar from a stitching perspective. We then consider more challenging setting where we ensure that the sender uses different information from the receiver, and also that its representations do not contain information relevant for the original receiver network. We do this by
2. Model Stitching
Model stitching has seen many variants (c.g. Bansal et al., 2021; Csiszárik et al., 2021; Hernandez et al., 2022) since it was first proposed (Lene& & Vedaldi, 2015). For simplicity, in this paper we only consider stitching between identical architectures and at matching points in the network. Let Agr denote the composition of all layers in the trained network A up to and including layer 1. Following Bansal et al. (2021),
2

---

Functional Alignment Can Mislead: Examining Model Stitching
we choose the stitching layer to be a randomly initialised 1 x convolutional layer preceded and followed by batch normalisation (loffe & Szegedy, 2015). Bansal et al. (2021) use convolutional layer as it has restricted expressivity. The untrained stitched model is therefore given by Byt o 8 Agi. We then train the stitched model by freezing the parts taken from the models A (the sender) and B (the receiver) and only optimising the stitching layer s. We then report the performance of the stitched model the test data. If the test performance of the stitched model is greater or equal to that of the receiver (which becomes the baseline), it is considered that models A and B are compatible at layer 1.
all instances of "1" have green background, etc. We refer to this dataset as "Correlated".
We use this problem to show that stitching is unable to dis- tinguish between models that use different features to make the prediction. To do so, we want to simulate a scenario where training various models leads to each learning a different pattern. One possible way of biasing the model towards picking up different rules is to modify the architec- ture. However, modifying the architecture means that we can't can't perform a one-to-one model comparison. For this rea- son, we choose to modify the training data instead as a way of biasing the models. We therefore create different variants of the dataset that lead to the model classifying based on either colour information alone, shape information alone, or different levels of relying on the combination between colour and shape.
In this paper we focus on ResNet-18 and ResNet-50 (He et al., 2016) models, but include results for VGG19 (Si- monyan & Zisserman, 2015) in Appendix D.I. Note that we stitch ResNet models before residual blocks and before the linear classifier. Therefore we refer to stitching before the first residual block as stitching "Res Block 1"; before the second residual block as "Res Block 2"; and stitching before the classifier as "Linear".
Importantly, note that we modify the data in such way that each model would still perform well (over 98% test accuracy) on the original Correlated dataset. This means that the models could have been trained on the original Correlated data if we had an appropriate way of biasing them in a controlled way through the training procedure alone. Below we provide brief description of the dataset versions created (see Figure 2a for visual examples). For full details on the creation of the datasets, see Appendix A.
Current stitching interpretation. This work is inspired by Hernandez et al. (2022) who find that stitching can reach high accuracy even when stitching from later lay- ers in Model A to much earlier layers in Model B. They believe that this could either be because the common intu- ition about how models process input is wrong, or because model stitching is able to match representations which are "different from what is expected". Nonetheless, they con- clude that functional similarity provides a meaningful way of comparing models. Their intuition is that "if two repre- sentations can be used for similar purposes then in some sense they encode similar information" (Hernandez et al., 2022). This agrees with Bansal et al. (2021) who argue "two networks with identical architectures, but very different in- ternal representations, would fail to be stitching connected"".
Colour MNIST (Correlated): The colour of the back- ground and class of the digit are correlated.
Digit with Uncorrelated Colour (Digit): Images contain- ing random combinations of background colour and digit. Target is given by the digit's class. Note that the colours were chosen from the same set of colours used to generate the Correlated dataset.
We argue that no meaningful conclusion can be drawn from analysing the stitch connectivity. When stitching is not suc- cessful, this could simply be because good enough map- ping between the representations was not found (Csiszárik et al., 2021). We claim that when stitching is successful, one cannot conclude that this is necessarily because the rep- resentations capture the same information. The next section shows the latter empirically.
Colour with Uncorrelated Digits (Colour): Images con- taining random combinations of background colour and digit. Target is given by the colour's 's class. Im- portantly, the model could learn to represent shape information, but this cannot help the model classify.
Colour with no Digit (Colour-Only); Images containing background colour and no digit. Target is the colour's 's class. The model cannot learn to base its predictions on shape information.
3. Inducing a Learning Bias Through the Data
We choose to compare models on a typical shortcut learning problem: a variant of colour MNIST (Bahng et al., 2020). In essence, colour MNIST adds colour as an additional correla- tion between the input variable and the target variable. We choose add colour fully-correlated background. For example, all instances of digit "0" have red background,
For clarity, note that the short version of the dataset's 's name (e.g. Digit) refers to the part of the input that is correlated with the label. We refer to the models by the name of the dataset they were trained on. Unless otherwise stated, the results are reported on ResNet-18 models. For full experimental details see Appendix B.
3

---

Functional Alignment Can Mislead: Examining Model Stitching
Corr.
6
7
7
8
1.000
125-
Stitched Model
Digit
6
6
7
7
T
Accurayy
-0.995
Fealture Rank
Correlated
Digit
0.990
Colour
Colour Colour
9
/
8
L
4
0.985
50-
Baseline
0.980
25
Only
Res Block Res Block
Res Block Res Block
Linear
Res Block 1 Res Block 2
Res Block
Pes Block
Linea
(a)
(b)
(c)
Figure 2. (a) Two example images for classes '6', '7', and '8' in each dataset type. (b) Stitching test accuracy when stitching at cach of the layers we consider, for all four types of model stitched into Digit model with Correlated data. Baseline is given by the test accuracy of the unstitched Digit model on the Correlated dataset, which is the task we assume to be given. Shaded area covers 100% of results range, markers show each result across multiple initialisations. We find that we achieve stitch compatibility across different layers for all the models we consider. (For each stitched Digit model, accuracy remains above its own unstitched baseline) (c) Rank analysis of the feature maps output by the receiver's first layer upon stitching. Shaded areas mark standard deviation.
4. Experiments
includes an uncorrelated digit), the model can only learn to represent solid colour information. Nonetheless, when stitching on the original task, we obtain test accuracy higher than that of the receiver alone, indicating stitching compatibility (see Figure C.
We start this section by adhering to Bansal et al. (2021)'s original setting. Namely, we assume we are given the task of classifying the Correlated dataset and we want to analyse the similarity of multiple models trained to solve this task. We show that stitching cannot distinguish between models capturing different biases in the data. To rule out the pos- sibility that the results we observed are due to information leak, we then move away from the original stitching setting (Section 4.2). We remove from the input (and as a conse- quence, from the sender's representations) any information related to the receiver's task and manage to achieve stitching compatibility.
Numerical rank. We analyse the rank of the feature maps after the first layer of the receiving network. This is a way of gauging the linear dependence of the sender's feature maps as seen through the lens of the receiver's ' representa- tion. If two sender representations have similar rank when processed by the receiver network, we cannot necessarily claim that they are perceived in a similar way. We can only say that the receiver represents them with a similar level of linear dependence. However, if they do not have the same rank, we take that additional evidence that the senders have learned different information according to the receiver. Details of the numerical rank estimation are provided in Appendix B.2.
4.1. Stitching Cannot Distinguish Between Different Biases
In this section we want to verify if stitching can help distin- guish between models that learn to classify using different information. We start by stitching various senders into the Digit model (i.e. the digit is correlated with the label and the background has a random colour). Note that for training and evaluating the stitch, we use the Correlated dataset as this is the data we assume we have access to, in accordance with the typical stitching procedure. All other datasets were created simply to train senders and receivers with different biases. We find that at all layers, all models we consider can easily be stitched into the Digit model (which becomes the receiver), obtaining a higher accuracy than that of the orig- inal Digit model (see Figure 2b). That is, even the Colour with Uncorrelated Digits model, which learned to classify based on colour information (ignoring shape), appears com- patible with the model that learned classify based on shape information only.
In Figure 2c we show the rank computed when stitching at each of the 5 layers we consider. We find that for stitching before the 2nd and 3rd residual blocks there is a clear gap between the rank of the processed representations of Digit with Uncorrelated Colour and those of Correlated, for exam- ple. This indicates that when fed into the receiving network, the representations of Digit with Uncorrelated Colour have a different degree of linear dependence from those of Cor- related and therefore are not perceived as equivalent by the receiver despite both being stitch-compatble.
The numerical rank of representations provides an estima- tion of their compression. Note that alternative estimators of compression could be considered. Each estimator, including numerical rank, comes with its limitations. Our objective is to include additional evidence that the various sender rep- resentations are not entirely equivalent when processed by the receiver. Following recent work (e.g. Masarezyk et al.,
Taking this further, we train model on patches of colour with no digit (' ('Colour-Only'.. As opposed to Colour (which

---

Functional Alignment Can Mislead: Examining Model Stitching
2023; Feng et al., 2022), we choose to use numerical rank to illustrate the difference in compression.
In light of these results, we argue that models that extract patterns from significantly different information can, in at least some cases, be easily stitched together. While it may be argued that the datasets chosen are too simple, it is known that SGDDttainned models find simple rules or shortcuts (e.g. Valle-Perez et al., 2018; De Palma et al., 2019), and there is no algorithm to decide when stitching is applicable. There- fore, models' stitching compatibility should not be taken to mean that they learn similar rules or that they capture similar information.
Arguably, in the stitching cases considered so far colour information could still "leak" through a model that did not learn to use it for classification. The same argument holds for structured shape information potentially leaking. We next perform two follow-up experiments: with a re- ceiver model trained only on shape information, we obtain full stitch compatibility on Colour-Only images (no digit is depicted, Figure 2a), despite no shape information be- ing present in the data; and stitching clustered, random noise simulating a sender network's 's representation (see Sec- tion 4.2) into the receiving network. These experiments strengthen our findings that models can be easily stitched to- gether even when they represent very different information.
Note that these last experiments (starting Section 4.2) de- liberately move away from Bansal et al. (2021)'s original setting. The stitch is now trained and evaluated on a differ- ent task to the one that the receiver was trained on. This is necessary in order rule out the effect of information leak. This extended setting allows us to emphasise that stitching alignment can abstract away from the input pat- terns that produced the sender's 's representations. Because of this, stitching cannot reliably distinguish between two models that capture different cues in the data. In the next section, we are going to explore this extended setting fur- ther by stitching together models that were trained to solve different tasks altogether.
4.2. Representations of Very Different Information Can Be Stitched Together
To check whether stitching between dissimilar models is due to the sender model "leaking" information expected by the receiver, we remove all information that could be expected by the receiver from the images we train and evaluate the stitch on. Concretely, we stitch Colour-Onie models into receivers trained on Greyscale MNIST data but this time using Colour-Only data to train and test the stitch. There no digit-like structure in the Colour-Only data which (if leaked) could be used by the MNIST receiver. The un- stitched MNIST models have baseline accuracy (when tested with MNIST data) of 99.0 + 0.1%; at every stitching layer, the Colour-Only sender increases this to 100% accu- racy, resulting in stitching compatibility. In other words, despite the sender representations not containing any digit information at all (as the input does not contain that infor- mation in the first place) we are able to successfully stitch those representations into a Greyscale MNIST model.
5. Other Successful Stitchings: Discriminative Case
So far we focused on simple problems which allowed us to control what information is available to the models. The remainder of the paper focuses on "harder" problems. We would like to highlight, however, that in such cases no clear ground truth exists and constructing arguments based on what "intuitively makes sense" can be deceptive. We simply offer these as supporting evidence to our well-controlled experiments on which all our claims are based.
Note that Bansal et al. (2021) consider the performance of the receiver as the baseline. In our previous experiments, although the sender and the receiver were trained on differ- ent datasets they were still trained to be compatible with the same task, which is Correlated (both the digit class and the colour class are correlated with the label). As a reminder, our goal was simply to induce strong bias towards solv- ing the task using either the digit or the background colour. Naturally, when tested on fully correlated data, both models were successfully solving the task.
Lastly, we stitch clustered, random noise (Clustered-Noise) into the receiver network. We create 10 random vectors to represent the mean of cach of the 10 classes. The vec- tors have the same dimensionality as the representations that would normally be expected by the stitching layer. For each class create 6K samples by adding random noise to the mean class vector, obtaining dataset of the same size as the original Correlated dataset (see Appendix A). Note that we do not use Clustered-Noise to train a sender. Instead, Clustered-Noiss simulates the representations of a sender network. We then stitch these simulated representa- tions onto the Digit receiver. We can achieve full stitching compatibility when using randomly generated represen- tations. As expected, the rank of these representations when fed into the receiver network is different from that of the learned representations (see Figure Cb).). We find similar results with VGG19 models (see Appendix D.I).
In a non-toy setting, when trying to stitch between models that are unlikely to represent the same input patterns, the receiver will inevitably perform poorly on the sender's task. This means that the baseline will usually be very weak. We choose to set a higher baseline, which is the performance of the receiver on its own task. For example, in the case when the ImageNet model is the receiver on Stylized data (which
5

---

Functional Alignment Can Mislead: Examining Model Stitching
Table Results of stitching various models into and from ImageNet. We highlight in bold the representations that were sticch-comaatibe. Note that "Block X" and "Linear" means that we are stitching before the Xth residual block and the classification layer respectively. Despite choosing more challenging baseline than typically considered for stitching, we are still able to find some models stitch-conin compatible at various layers. * *Accuracy results for this baseline are given by Stylized model's performance atypical, are higher than Stylized model's performance on the Stylized task (56.18% top-1 and 78.96% top-5 accuracy). Note that with the weaker bascline, we find compatibility at all layers.
ImageNet data which, although
Block
Block 2
Block 3
Block 4
Block 5
Linear
Baseline
ImageNet Stylized
Top-1 Top-5
58.39
58.61
61.53
65.31
70.43
74.34
60.18*
80.57
80.61
83.28
86.13
89.78
91.93
82.60*
Stylized to ImageNet
Top-1 Top-5
10.61
11.23
17.60
30.26
45,45
49.17
76.13
22.65
23.08
34.34
52.52
69.57
72.30
92.86
10-class ImageNet to Birdsong
T Top-5
15.40
19.00
25.00
46.20
75.60
88.40
77.98
63.80
68.40
76.40
88.60
98.60
99.80
96.07
Clustered-Noise
Top-1 4.77
4.11
96.59
100
100
100
76.13
to ImageNet T
13.51
11.99
99.77
100
100
100
92.86
we introduce in Section 5.1), the weak baseline would be the performance of the ImageNet model on the Stylized data (16.4% top-5 accuracy). We choose the more compet- itive baseline, which is the performance of ImageNet on ImageNet data (92.86% top-5 accuracy). There is only one exception to this choice of baseline, which we emphasise when discussing the experiments. For completeness, we include the weaker baseline in Table F.1 but show the more challenging setting in the main body of the paper.
which is close to the accuracy we achieve when stitching. Note, however, that we achieve stitching compatibility ac- cording to the alternative baseline (receiver's performance on the sender task), which in this case is only 16.4% top-5 accuracy. Nonetheless, choose consider the more chal- lenging baseline and report stitch incompatibility instead. On the other hand, we find that we can achieve stitch- compatibility at several different layers when stitching ImageNet into Stylized (see Table 1). We see this as ad- ditional evidence that there are cases in which functional alignment methods such as stitching cannot distinguish be- tween models with different biases.
Note that since for most of the experiments in this section we are using a pretrained ImageNet model, we only do one run of the experiments. All discriminative models considered in this section are ResNet-50. All stitches in this section are trained for 20 epochs using learning rate of 10-1 and we do not perform any hyperparameter tuning for training the stitches (full training details in Appendix B.4). All results from Section up to and including Section 5.4 are collated together in Table 1.
5.2. Clustered-Noise
Following the same procedure as in the case of stitch- ing Clustered-Noise MNIST-trained models, stitch Clustered-Noise to a model pretrained on ImageNet. Even with no hyperparameter tuning, we achieve stitch compati- bility after the second residual block (see Table 1).
5.1. Shape and Texture Bias
5.3. Different Extracted Information
Geirhos et al. (2019) showed that standard-traine ImageNet models are biased towards learning texture information. They then constructed Stylized ImageNet (Stylized), a ver- sion of the ImageNet dataset where local texture is no longer predictive, forcing models to focus on shape information (less localised patterns).
We next use one dataset and stitch such that we remap its classes. This approach uses just one model and one dataset, changing fewer variables than some other experiments. We train a model on greyscale-Man -NIST, stitch it back together at a chosen layer, and train the stitch on MNIST data but with labels offset in a cireular shift. The effect of this is that the sender's representation of, say, digit '0" is stitched so that the receiver classifies it as class 'I'. Yet, an un- stitched model trained on MNIST and presented with an image for '0' must represent different information from the same model presented with a '1' otherwise it could not discriminate. Equivalently, in the Colour-Only dataset, red would now be classified as green (see Figure A. This
As a real-world version of our previous shape and colour experiment, we take pretrained ImageNet and Stylized mod- els and aim to stitch them together. Note that we cannot successfully stitch Stylized into ImageNet. As we will dis- cuss in more detail in Section 7, this is likely due to Stylized model's 's poor performance on the Stylized task. Stylized only achieves 78.96% top-5 accuracy on the Stylized dataset,
6

---

Functional Alignment Can Mislead: Examining Model Stitching
1.0
Class Mapping We match classes across datasets. For ex- ample, with Fashion-MNIST and MNIST, we might pair classes like 'Bag' "0', 'Coat' etc. Within each class pair, we match images randomly. The loss is the 1.2 distance between the generated image and the corresponding paired image from the other dataset.
Test Accuracy
Class offset 1
0.9
Correlated
Colour-Only
BW
0.8
Res Block
Res Block 2 Res Block 3 Res Block 4
Linear
Embedding Mapping We match embeddings in the latent space of the autoencoders. We match the stitched em- beddings of the sender against those of the original encoder of the receiver and solve this as a linear sum assignment problem. The loss is then given by the sum of squared distances between mapped embeddings. can effectively be considered that we create joint dataset where we pair up each training sample from dataset A (used to train encoder A) with another train- ing sample from dataset B (used to train encoder B). Training samples are paired up using the encodings (i.e. in the embedding space). Specifically, we solve the linear sum assignment problem in the autoencoders' bottleneck to pair up samples from datasets and B, Passing an image from dataset through encoder A, the stitch is trained to map it to its corresponding sam- ple from dataset B, passed through encoder B.
Figure 3. Test accuracy when stitching model to itself with the dataset the model was trained on, remapping class labels with an offset of 1 (e.g. an image of 'T' is labelled class '8"). Base- line performance of Correlated and Colour-Only was 100% while BW (groyseale-NNIST) was 99%, shown as green dotted line. Shaded regions show full range of results. We again find stitch compatibility, especially in the later layers.
experiment was successfully repeated for models trained on Colour-Only and Correlated datasets (see Figure 3). The same models can be restitched with different shift offsets and at different layers (see Appendix E for more details).
5.4. Different Tasks and Modalities
We then attempt stitch a model trained recognise bird songs pretrained ImageNet model. For this we use 10-class version of the xeno-canto (Vellinga & Planqué, 2015) dataset. We use publicly available code to create a version of the dataset with 10 bird classes, preprocess the data into spectrograms and train a ResNet-50 to solve the task (full training details in Appendix B.5). The model achieves 77.9 77.8% accuracy on the birdsong recognition task.
For full experimental details, see B.3. Following Moschella et al. (2022), we perform a qualitative evaluation of the generative stitch. We are able to reconstruct MNIST-like digits from Fashion-MNIST test data using both mappings we consider (see left-hand side of Figure 4), with clearer im- ages obtained for the Embedding Mapping. We then stitch CIFAR-I0 (Krizhevsky, 2009) encoder onto an MNIST decoder (see right-hand side of Figure 4) and once again obtain depictions of MNIST- like digits.
To keep in line with the typical stitching regime, we restrict the ImageNet dataset to a 10-class version. For simplicity, we choose the first 10 classes of ImageNet. In this case, we once again find we can successfully stitch models, yet their representations are not semantically similar (i.e. do not capture the same input patterns).
Although this setting differs significantly from Bansal et al. (2021)'s original stitching proposal, it provides evidence that more broadly representations can be aligned without necessarily representing the same information.
6. Stitching Other Embeddings: Autoencoders
7. Discussion and Observations
Following Fumero et al. (2024) and Moschella et al. (2022), we consider the case of mapping representations of autoen- coders. Moschella et al. (2022) extend the definition of model stitching to the generative case by stitching two au- toencoders trained on MNIST from different initialisations. Their intuition is that the two models should be compatible since they are simply trained from different initialisations.
We believe building tools to meaningfully compare neu- ral networks is an important objective in deep learning re- search. Such tools are treated as potential ways of better understanding neural networks and therefore can be seen as part of the Explainable AI puzzle. For example, alignment between models has been used in the attempt to better un- derstand architectural differences (e.g. Nguyen et al., 2020; Raghu et al., 2021), aspects of continual learning (Ramasesh et al., 2021; Kim & Han, 2023), or training with different types of data (e.g. McGuire et al., 2023), or even factors that contribute to networks' alignment with human percep- tion (Demirean al., 2024).
We train one autoencoder on MNIST and another one on Fashion-MNIST (Xiao et al., 2017). We take the encoder of Fashion-MNIST and the decoder of MNIST and stitch them together. We consider two approaches to stitching between inputs and reconstructions of two different datasets:
7

---

Functional Alignment Can Mislead: Examining Model Stitching
Input
Embedding
9
5
8
2
9
8
6
3
3
4
G
2
1
9
2
9
6
8
reconstr.
Class
4
8
9
6
9
5
8
9
reconstr.
Figure 4. Examples of reconstructions when stitching autoencoders trained on different datasets. Left: Fashion-MNIST. Right: CIFAR- 10. Top to bottom: Inputs given to the stitched autocncoders, reconstructions obtained with the Embedding Mapping method, reconstructions obtained with the Class Mapping method. MNIST-like digits are generated from Pashion-MNIST and CIFAR-10 inputs.
The objective of our paper is to more closely analyse one such tool proposed to measure similarity and alignment. From this perspective, our work fits within the literature exploring the limitations of representation comparison tools (e.g. Ding et al., 2021; Davari et al., 2023). Most related to our work but in the context of Representational Similarity Analysis (RSA), Dujmović et al. (2022) construct artificial datasets to argue that RSA cannot distinguish between mod- els with "different feature encodings".
clustered (but not necessarily using the same information to assign samples to a cluster). While this is a straightfor- ward observation, we include an experiment to intuitively illustrate this. We take our Clustered- experiment in Section 4.2 and increase the level of noise around each clus- ter. As we do so, the separability decreases and, as result, the stitching accuracy drops (see Table D.3).
Therefore, we believe our experiments cast a shadow on the interpretation of functional alignment. While it might be possible that models converge to a shared understanding of reality, we believe our results show that one needs to look beyond functional alignment to support this claim. We urge the community to rethink the belief that mapping between two clustered representations means that models capture the same aspects about the world. Models that converge to a shared statistical model of reality might be functionally compatible, but functionally compatible models do not necessarily capture the same patterns in the input.
Through a range of experiments we have shown that we can stitch representations used to solve different tasks using dif- ferent modalities. Alignments between different tasks (e.g. Gygli et al., 2021) and even different modalities (e.g. Rad- ford et al., 2021; Merullo et al., 2023; Koh et al., 2023) are not new and have been exploited before, especially between text and images in the context of image captioning.
To explain this type of alignment, which is typically seen as an alignment of "semantic concepts", Huh et al. (2024) pro- pose the Platonic Representation Hypothesis. This hypothe- sis states that models "are converging to shared statistical model of reality in their representation spaces" which makes alignments such as stitching possible. While in the context of image captioning this argument intuitive, Huh et al. (2024) take this hypothesis further and argue that represen- tation alignment more generally, including across different tasks and modalities, is explained by representational conver- gence. This echoes the Anna Karenina Hypothesis (Bansal et al., 2021) stating that "all successful models end up learn- ing roughly the same internal representations". Importantly, note that Huh et al. (2024) simply use functional alignment as supporting evidence for their hypothesis, but primarily focus on structural alignment.
Lastly, we believe controlled experiments where we can create models that learn to solve a task using different input cues are important for evaluating model comparison tools more generally. We regard these as necessary for establish- ing the validity of hypotheses around both functional and structural similarity.
Can anything be stitched onto anything? No. We have seen throughout the paper that some layers and even some models (Section 5.1) cannot be successfully stitched to- gether. There are several potential reasons for this. In some cases, this could be a matter of learnability of the stitch. Note that all the experiments in Section 5 were run with the same learning rate, we did not do any hyperparameter tuning and we only trained the stitch for 20 epochs. In the case of stitching Clustered-Noiss into the ImageNet model for example, it might be the case that with slightly longer training, we would be able to successfully stitch earlier lay- ers. This highlights another limitation of stitching which is that when models cannot be stitched together, it is un- clear whether this is because a good linear mapping was not found or because the representations are not linearly mappable. This is known issue with model stitching (e.g. Hermandez et al., 2022).
Are our results possible because models reached shared understanding of reality? No, we strongly disagree that this is applicable here. We have shown that alignments can be found between ImageNet models and birdsong recog- nition models, or even Clustered-Noise. It is difficult to imagine how there could be a shared understanding of real- ity between these three datasets.
We believe a more plausible hypothesis is that we were able to align the representations largely because they were well
8

---

Functional Alignment Can Mislead: Examining Model Stitching
The second reason why we were not able to stitch is because of the difference in task difficulty and/or original model performance and, as a result, achievable separability in the representational space. For example, in the case of stitching Stylized to ImageNet, the Stylized model achieves signifi- cantly lower performance on the Stylized dataset than the ImageNet model on the ImageNet dataset. So for the prob- lems that the models were trained on, there is a big gap in performance. This could be because Stylized might be harder problem to solve altogether but also because as Geirhos et al. (2019) mention, they did not necessarily train their Stylized model to achieve competitive performance on the Stylized data. It might be the case that with a better- performing model on Stylized we would be able to stitch onto ImageNet but this is beyond the scope of the paper.
Where next? We believe that meaningfully compare rep- resentations we need to be able both gauge what informa- tion they are encoding and how that information is "mapped out" in the embedding space. We have demonstrated that the functional perspective alone, and stitching in particular, cannot do that. Capturing these is of course not straight- forward and the lack of ground truth makes this even more challenging. We believe the field would benefit from care- fully constructing controlled benchmark problems rather than advancing based on what is "intuitively expected". Our work proposes a simple scenario for evaluating and reason- ing about representational differences and similarities but it is by no means comprehensive enough for constructing sufficiently varied baseline more generally. Nonetheless, it provides a starting point for such a framework which we believe is of paramount importance if we aim to create tools that capture meaningful differences between models.
Should stitching then be used quality? While there might be cases in which stitching is insightful, we believe it is difficult to think of what stitching can tell us about model quality that existing methods don't 't do already. If we cannot successfully stitch because of an incompatibility in task performance, this can already be seen by simply looking at the models' accuracy on the task. If two models cannot be stitched because the receiver does not cluster the representations sufficiently well at intermedi- ate layers, then we can simply look for direct measures of representational clustering.
measure of model
8. Conclusions
We showed that networks can use or represent very different information, yet classify samples with similar accuracy. Im- portantly, the different representations can easily be stitched together. This leads to question the usefulness of study- ing models' purely functional similarity, and in particular their stitching compatibility, to determine whether or not they capture similar information. Being able to distinguish between models that make decisions based on different in- put patterns is important, especially given the propensity of deep learning models to learn shortcuts. We hope that our work encourages the community more carefully interpret the results of model stitching, and understand what it is ac- tually responding to. We also hope our work will encourage researchers to focus on creating model comparison tools that can reliably capture informational similarity. We propose artificial shorteut learning problems as a starting point for reasoning about this in a controlled way.
Finally, one of the reasons why stitching was proposed and adopted as an alternative to methods like CKA is because it allows us identify when a representation is "better" than another, rather than simply "different". We challenge the understanding that achieving higher stitching accuracy than the baseline (receiver's 's own accuracy) means the representa- tion of the sender network is better than that of the receiver. To this end, we simply swap the receivers and the senders considered in the previous experiments. In Figure 2b we observed a higher test accuracy for Colour and Correlated compared to the Digit baseline, which would be taken to in- dicate that their representations are "better" for discriminat- ing samples on the problem we consider. However, stitching Digit (as sender) into Colour or Correlated (as receivers) also leads to higher accuracy for particular layers compared to the baseline (see Table C.I and Appendix C.1), which is a contradiction. Finally, we stitch model with itself (Digit vs Baseline) and obtain an increase in accuracy (see Fig- ure C.) well change in the rank of representations (see Figure Cb).). This further indicates that although the stitching layer has reduced expressivity compared to a fully- connected stitch, it still cannot be claimed that it simply aligns the representations of the sender and receiver without any additional processing. Therefore, given the contradic- tory results, we do not think stitching accuracy is suitable for comparing the quality of models' representations either.
A Acknowledgements
The authors acknowledge the use of the IRIDIS X High Per- formance Computing Facility, the ECS Alpha Cluster, and the Southampton Wolfson Al Research Machine (SWARM) GPU cluster generously funded by the Wolfson Foundation, together with the associated support services the Univer- sity of Southampton in the completion of this work. H.M. is funded by PhD studentship provided by the School of Electronics and Computer Science. The authors would like to thank the members of the VLC research group for useful discussions, feedback, support, snacks, and good fun.
9

---

Functional Alignment Can Mislead: Examining Model Stitching
Impact Statement
Cho, K., and Oh, A. (eds.), Advances in Neural Informa- tion Processing Systems, volume 35, pp- pp- 33054-33065, 2022.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Fumero, M., Pegoraro, M., Maiorca, V., Locatello, F., and Rodolà, E. Latent functional maps: a spectral framework for representation alignment. The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
References
Bahng, H.,, Chun, S., Yun, S., Choo, J., and Oh, J. Learn- ing de-biased representations with biased representations. In International Conference on Machine Learning, 2020.
Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wich- mann, F. A., and Brendel, W. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations. 2019.
Bansal, Y., Nakkiran, P., and Barak, B. Revisiting model stitching to compare neural representations. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 225-236, 2021.
Geirhos, R., Jacobsen, J.-H., Michaelis, C.,, Zemel, R., Bren- del, W., Bethge, M., and Wichmann, F. A. Shortcut learn- ing in deep neural networks. Nature Machine Intelligence, 2(1 1):665 1:666-673, 2020.
Csiszárik, A., Körösi-Szabó, P., Matszangos7, A., Papp. G., and Varga, D. Similarity and matching of neural network representations. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, vol- ume 34, pp. 5656-5668, 2021.
Gygli, M., Uijlings, J., and Ferrari, Towards Reusable Network Components by Learning Compatible Represen- tations. Proceedings of the AAAI Conference on Artificial Intelligence, 35(9):620-77229 May 2021. ISSN 2374- 3468. doi: 10.1609/aaai www... Number: 9.
Davari, M. R., Horoi, S., Natik, A., Lajoie, G., Wolf, G., and Belilovsky, E. Reliability of CKA similarity measure in deep learning. In International Conference on Learning Representations, 2023.
He, K., Zhang, X., Ren, S., and Sun, Deep residual learning for image recognition. In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016. doi: 10. N..1109/VVR.2010
De Palma, G., Kiani, B., and Lloyd, S. Random deep neu- ral networks are biased towards simple functions. In Advances in Neural Information Processing Systems, vol- ume 32, 2019.
Hernandez, A., Dangovski, R., Lu, P. Y., and Soljacic, M. Model stitching: Looking for functional similarity be- tween representations. SVRHM 2022 Workshop @ NeurIPS, 2022.
Demircan, C., Saanum, T., Pettini, L.., Binz, M., Baczkowski, B., Doeller, C., Garvert, M., and Schulz, E. Evaluating alignment between humans and neural network representations image-based learning tasks. Advances in Neural Information Processing Systems, 37: 122406-122433. 2024.
Huh, M., Cheung, B., Wang, T., and Isola, P. Position: The platonic representation hypothesis. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 20617 20617-20642. PMLR, 21-27 Jul 2024.
Ding, F. Denain, J.-S., and Steinhardt, J. Grounding representation similarity through statistical testing. Ad- vances in Neural Informaiin Processing Systems, 34: 1556-1568, 2021.
loffe, S. and Szegedy, C. Batch Normalization: Accelerating deep network training by reducing internal covariate shift. Proceedings of the 32nd International Conference on Machine Learning, pp. 448-456. PMLR, June 2015.
Dujmović, M., Bowers, J. S., Adolfi, F., and Malhotra, G. Some pitfalls of measuring representational similarity using representational similarity analysis. bioRxiv, pp- 4022-04.22 2022.
Kim, D. and Han, B. On the stability-plasticity dilemma of Blass-incremental learning, In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 20196-20204 2023.
Feng, R., Zheng, K., Huang, Y., Zhao, D., Jordan, M., and Zha, Z.J.. Rank diminishing in deep neural networks. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D.,
Klabunde, M., Schumacher, T., Strohmaier, M., and Lem- merich, Similarity of neural network models: survey
10

---

Functional Alignment Can Mislead: Examining Model Stitching
of functional and representational measures. ACM Com- put. Surv. 57(9), May 2025. ISSN 0360-0300. doi: 10.1145/3728458.
deep learning dynamics and interpretability. Advances in Neural Information Processing Systems, 30, 2017.
Raghu, M., Unterthiner, T., Korbblith, S., Zhang. C., and Dosovitskiy, A. Do vision transformers see like convolu- tional neural networks? Advances in Neural Information Processing Systems, 34: 121 12-1222,,,, 2021.
Koh, J. Y., Salakhutdinov, R., and Fried, D. Grounding language models images for multimodal inputs and outputs. In International Conference Machine Learn- ing. pp. 17283-17300. PMLR, 2023.
Ramasesh, V. V., Dyer, E., and Raghu, M. Anatomy of catastrophic forgetting: Hidden representations and task semantics. In International Conference on Learning Rep- resentations, 2021.
Kornblith, S., Norouzi, M., Lee, H., and Hinton, G. Sim- ilarity of neural network representations revisited. In Proceedings of the 36th International Conference on Ma- chine Learning, pp. 3519-3529. PMLR, May 2019.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z.,, Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Viston (IJCV), 115(3):211252, 2015. doi:
Krizhevsky, Learning multiple layers of features from tiny images, January 2009.
Lene, K. and Vedaldi, A. Understanding image representa- tions by measuring their equivariance and equivalence. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 991-999 2015.
10.1007/s1 126-77/6163----1--88166633
Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. In Interna- tional Conference Learning Representations, 2015.
Masarczyk, W., Ostaszewski, M., Imani, E., Pascanu, R., Mitoš, P., and Trzcinski, T. The Tunnel Effect: Building Data Representations in Deep Neural Networks. Ad- vances in Neural Information Processing Systems, 36: 76772-76805, December 2023.
Sucholutsky, I., Muttenthaler, L., Weller, A., Peng, A., Bobu, A., Kim, B., Love, C., Grant, E., Groen, I., Achterberg. J., et al. Getting aligned on representational alignment. arXiv preprint arXiv:23100 10. 2023.
McGuire, S., Jackson, S., Emerson, T., and Kvinge, H. Do neural networks trained with topological features learn different internal representations? NeurIPS Workshop on Symmetry and Geometry Neural Representations, pp. 122-136. PMLR, 2023.
Valle-Perez, G., Camargo, C. Q., and Louis, A. A. Deep learning generalizes because the parameter-functe map is biased towards simple functions. arXiv preprint arXiv: 1ativ/805.. 01800088222 2018.
Merullo, J., Castricato, I.,, Fickhoff, C., and Pavlick, E. Lin- early mapping from image to text space. In The Eleventh International Conference on Learning Representations, 2023.
Vellinga, W.-P. and Planqué, R. The Xeno-canto collection and its relation to sound recognition and classification. In Cappellato, L., Ferro, N., Jones, G. J. F,, and SanJuan, E. (eds.), CLEF (Working Notes), volume 1391 of CEUR Workshop Proceedings. CEUR-WS.Co SOUR-WS.org. 2015.
Moschella, L., Maiorca, V., Fumero, M., Norelli, A., Lo- catello, F., and Rodolà, E. Relative representations enable zero-shot latent space communication. International Conference on Learning Representations, 2022.
Xiao, H., Rasul, K., and Vollgraf, R. Pashion-MNIST. novel image dataset for benchmarking machine learning algorithms, 2017.
Nguyen, T., Raghu, M., and Korbllihh, S. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. In International Conference on Learning Representations, 2020.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. $748-8763 PMLR, 2021.
Raghu, M., Gilmer, J., Yosinski, J., and Sohl-Dickstein, Sveca: Singular vector canonical correlation analysis for

---

Functional Alignment Can Mislead: Examining Model Stitching
0 1
2
3
4
5
6
7
8
9
Figure A.I. Colour swatches for the base colour_map. Digits font-based labels here, not MNIST images
A. Datasets
The following variants were created:
MNIST
This is simply MNIST data (monochrome handwritten digits) expanded to channels and normalised.
There is target-dependent colour information for the model to learn, expect it recognise the digits based features such shape, or the amount of white, the texture of the white-grey- edges. It cannol learn to colours to recognise classes these are never present.
It is possible that kernel weights will learn rely on the three colour channels always being equal.
Colour MNIST (Correlated) We wanted to provide an opportunity for some semantically different representations to be learned. this case, having the background colour correlate with the target digit being displayed.
We expect that the models trained on Correlated data will mostly learn to rely on the colour. However, they may learn the shapes of the digits, or the amount of white caused by the digit. They may rely on pixels always being at one of two rgb values (white/colour) in any one image, never an intermediate value. They may learn to rely entirely on a small number of pixels that never white e.g. corner pixels.
To generate the Correlated dataset, we used the colour -NIST package (Bahng et al., 2020) which uses MNIST data, but changes the colour of the background depending on the digit being represented (Figure A.1). The package snaps any non-zero pixels to white ([255,4) 255, 255]). This could be thought of as changing the greyscale information to binary, and indeed reducing the information content as a result. The background colour is changed from black (IO, 0, 01) to one of 10 values, always matching the data label.
For all of these datasets, to avoid the model learning something as simple specific RGB values, the base RGB background values are modified by random 10% per image (such that the colour is always flat, but will vary slightly from image to image within a class). The same variation is injected into the test datasets. This is analagous to requiring that colour's learned in a generalised way.
Digit with Uncorrelated Colour (Digit) To encourage models to learn representations which recognise digits, but cope with different background colours, we created a dataset with digits over laid on backgrounds whose base colours were randomly selected. i.e. the digit does not correlate with the colour.
We expect models trained on Digit to learn features like shape. Assuming the randomisation is sufficient, they cannot learn to rely on colour. The choice of base colours and inclusion of colour variation mean that the Digit models cannot learn rely on a single colour channel or pair of colour channels, and mus! learn to tolerate a range of colours being used. Nonetheless, they may learn to rely on any all pixels in an image being one of two colours.
Colour with Uncorrelated Digits (Colour) To encourage representations of colour information, but which are able to tolerate the presence digits, we generate a dataset in the same way as for Digit but in which the background colour (not the uncorrelated digit) is the target.
Models trained on Colour camnot use the digits to classify because they are uncorrelated with the background colours. Successful Colour models may rely on colour being sampled in specific locations which never contain digit pixels (e.g. corner), or might learn to average across the image. They may learn rely on the presence of some white pixels as the dataset does not contain any solid-colour images. But they may not be generally immune to non-digit-like patterns of white pixels.
Background Colour Only (Colour-Only) To force the learning of different representations which cannot be related to shape, size, or edge-effects of digits (and which may not be able tolerate those features), the 'Colour-Only' dataset
12

---

Functional Alignment Can Mislead: Examining Model Stitching
Figure A.2. Simulated "Clustered-Noise data sample for ResNet-18 at Res Block 2, showing 64 7x7 feature maps. If this represented Class all other Class data instances would be noisy versions this.
was created in which no digit image is present, only solid background colour (plus 10% variation). As in 'Colour' (see Figure 2a), the base colour is the target.
The model may learn to rely on the colour in only one region of the image, or to assume the colour is constant everywhere.
Clustered-Noise (Noise) To produce high- representation, without relying on specific, image-related features, we generate synthetic datasets in the form Clustered-
Passing an image through the first layers of a sender model produces a set of activations specific to the final layer before the cut. For example, with ResNet-18 cutting just before Res Block 2 of the receiver, the sender output will be 64 channels of size 7x7 (Figure A.2.) The synthetic dataset must be matched in shape to the layer being stitched As such, it has to be regenerated depending the layer in question. It does produce images, rather synthetic activations. Each data instance is represented by a point in activation space based on its class, plus random offset vector. Thus each class is located in cloud e-ball. For example, in the Res Block example, each of the ten classes will be centred around different random point in 3136-dimensions.
There is no attempt to create structure within or between the feature maps, and it is not derived from actual activation data. The high-dimensionality means it is likely that the clusters of data points will be highly separable, though this not enforced and has not been verified.
ensure the same base representations are used for train and test datasets for a given layer or test, a latrrat_aacivvvie and accessed via dataloader.
function was created (Alg 1). A train and dataset can then be generated (Alg 2)
B. Experimental details
B.1. Stitching Between Models
In an extension of the experiments by Bansal et al. (2021), we train models on different special datasets (Section A) and then stitch between them, assessing the change in accuracy.
Hyperparameters for Model Training batch size=128, 4 Epochs, SGD, Ir=le-1, momentum=0.9, weight decay=le-4.
13

---

Functional Alignment Can Mislead: Examining Model Stitching
Algorithm 1 Generate Base Activations
Procedure GENERATE GENERATE, AAT.AAATITIIINSSStmmcsss..e. representationnssaaee
for c 0 to num 1 do
activations(c + rand[represennaaiionsaaad
end for return activations
End Procedure
Algorithm 2 Generate a dataset (unshufiled)
Procedure SYNTHETTCDAATASTT((rin,,, activations, noise) if train then SamplesPerClass 6000 else
SamplesPerClan +- 1000 end
for c 0 to num classes 1 do
datalc* * SamplesPPrCllas: : ((c+ + 1) SamplessPerClass]] + activations(c
targetslc* SamplesPerClan : ((c+ + 1) SamplesPPeClass)) t
end for
data data + noise randn
data + clampídata, 0,
End Procedure
Hyperparameters for Model Stitching batch_size=128, 10 Epochs, SGD, Ir=le-4, momentum=0.9, weight Hyperparameters for Noise Stitching batch_size=64, Epochs, SGD, Ir=le-4, momentum=0.9, weight.decay=le-2
Two versions of this performed, one stitching from each different model into the Digit with Uncorrelated Colour receiver model, and the other stitching from that model as sender into each of the others. This will allow us to assess the symmetry of the stitching process. Recall that models trained on each of the datasets represents a model that could have arisen naturally by training on Correlated data, but with the advantage that we know something about features that may (or cannot) be learned.
In this experiment, all image datasets used to train models. Each of the models (including Digit) is then stitched into the Digit model at each of 5 points: (e.g. see Figure B.1).
Cut before Res Block
Cut before Res Block 2
Cut before Res Block 3
Cut before Res Block 4
Cut before Linear Layer
The accuracy and rank are then measured using the Correlated test dataset. Accuracy and rank are also measured for the whole, unstitched 'Digit' model using the Correlated test dataset for comparison. This test configuration is different from that used by Bansal et al. (2021) and Hernandez et al. (2022). They used the same dataset for training and testing their models. We wanted to examine the case where:
1. The dataset for training and testing the stitch is correlated (biased) ie. it contains class-correlated features other than the intended learning target (the written digit): in general this will be the case, even if unintentionally. Recall that the bias is that the background colour is correlated with the labelled digit.
2. The sender models are likely have learned different features from the receiver.
14

---

Functional Alignment Can Mislead: Examining Model Stitching
Training "Colour" model
maxpool
Res Block
Auggoool to Linear
"Colour"
Block
Block.
Training "Digit" model
Canvil maxpool
Res Block
Res Block 4
Avgpool to
"Digit"
Block
Block
dataset
"Colour" sender
"Diglt" recelver
maxpooi
Convi to
Siitch
Res Block
Avgpoiol to Linear
"Correlated" dataset
Black
Block
Block
Measure
Rank
Training Evaluating Stitch on "Correlated" data
Figure B.I. To illustrate the process, prior any stitching, model is trained Colour dataset and another the Digit dataset. Because will be used receiver, the Digit model is then evaluated on the Correlated dataset as haseline, without any stitch layer. For stitching, the Correlated dataset used: the train dataset for training the stitch, and the dataset for evaluating functional performance. In this example, the Colour model is the sender and the cut is before Res Block 2. The Digit model is the receiver and the stitch layer connects just after Res Block Note that the Rank is measured the first conv layer of the Digit receiver model
15

---

Functional Alignment Can Mislead: Examining Model Stitching
"Clustered "Nisee"
dataset
Stitch
Res Block
Res Block 2
Res Block
Res Block 4
Avgpool to Linear
Convil to maxpool
"Digit" model
Measure Rank
Figure B.2. In this example, the Digit model is the receiver and the cut before Res Block 2. Note that the Rank measured at the first conv layer of the Digit receiver model. The shape of the 'Clustered-Noise data must match the receiving layer.
B.2. Stitching from Clustered-Noise
train ResNet-18 models for each the image-based datasets described. For each model, the accuracy recorded for the associated test dataset. The model cut before the first ResNet block, and prepended with the stitch to create stitch + receiver model. The stitch (only) is then trained the Clustered-Noise dataset.
The stitch receiver model (Figure B.2) is then tested against the synthetic test dataset. This is repeated, cutting before each Res Block (a layer comprises 2 Basic Blocks. The skip connections are preserved), and before the final fully-connected linear layer. We also record the rank of the activations for the Clustered-Nowd test dataset at the first convolutional layer in the receiver model after the stitch. This to provide insight into how the stitched data is "perceived" by the different models. Also how that compares to when dataset images are processed by the model. For each test at each stitch point the synthetic dataset was regenerated reduce the likelihood that results are due one randomly selected of points having a significant structure.
B.2.1. EXAMINING THE RANK
To gain further insight into how the representations being sent are "perceived" by the receiver, we analyse the rank at the first receiver layer. Note, however, that generally analysing the rank of two sets of representations cannot tell us anything meaningful about their informational similarity. We are simply arguing that the receiver cannot map the various sender representations in entirely equivalent way if the rank in the first receiving layer different.
1. Train several models each on a different, but related dataset. For example, Colour MNIST can create different dataloaders of MNIST digits on coloured backgrounds where the colours correlate or not correlate with the digit class.
2. Stitch models to each other such that sender from layer to L, and receiver is from layer L+1 to Classifier. Train the stitch.
3. Collect the representations at layer L+1 obtained using the test dataset. For estimating the rank, we follow Masarczyk et (2023), who compute singular values of the sample covariance matrix and threshold these at 1e-3 the largest singular value.
B.3. Stitching Autoencoders
We tested whether two autoencoders, trained on different datasets, could be stitched at their bottlenecks. In our experiments the stitching is placed immediately after the encoder. Autoencoders are typically trained using reconstruction loss, usually
16

---

Functional Alignment Can Mislead: Examining Model Stitching
based LI or L.2 distance between input and output images. However, when the input and output images come from different datasets, reconstruction loss becomes inappropriate since the images won't match. To address this, we two methods for learning to map images from one dataset to another, we refer to these approaches as class mapping and embedding mapping.
Class Mapping When datasets have class labels, we can match corresponding classes across datasets. For example, with CIFAR-10 and MNIST, we might pair classes like 'Airplane" '0' 'Horse' ''', etc. Within each class pair, we match images randomly. During training, an image from one dataset is passed through its encoder, then through the stitch, and finally through the decoder corresponding the other dataset to generate an image. Our loss function is then the L2 distance between the generated image and the image that has been paired with our input image.
Embedding Mapping Alternatively we can match points one-to-one in the latent space of the autoencoders and train the stitch to map points from one latent space onto points of the other. This involves first encoding batches from both datasets through the respective encoders, producing features 6 and 62- The encodings e1 are then passed though the stitch, producing features We then match points to points E2 in a one-to-one manner such that the sum of the distances is minimised. This mapping challenge is known as the linear sum assignment problem, which can be efficiently solved using SciPy. Once this mapping has been found, our loss function is simply the sum of squared distances between paired points in s and C2- The full calculation of this loss is detailed in Algorithm 3.
Algorithm 3 One-to-One Autoencoder Stitching (Loss Calculation)
Procedure CALCULATE LOSS(AF,, AF2, stitch, dataset1, dataset2)
dj dataseti
d2 ~ datasetz
el < AFq. enccoderd(d)
c2 + e.g.dercd
s + stitch(ei)
cost_matrix + Pairuise 02)
i,j < TineerSum.Asiigmeennssss.oo
return Σ [a
End Procedure
We trained autoencoders to map Fashion-MNIST to MNIST and CIFAR-10 to MNIST using these techniques. The architecture of our encoder includes convolutional layers, ReLU activations, and max pooling layers for downsampling. Following this, features are flattened and passed through a series of linear layers. The decoder follows the same structure but in reverse. The bottleneck is a vector of length 128 for all experiments. All parameters were frozen except for the stitch, which connected the encoder of one model to the decoder of the other. All autoencoders were trained for 25 epochs with a learning rate of le-4 using the Adam optimiser. Embedding Map stitching was trained for 25 epochs using SGD with a learning rate of le-5, momentum of 0.9, and weight decay of 0.01. The class mapping stitching was trained for 20 epochs using SGD with the same parameters except the learning rate set to le-2. The learning rate and total epochs for each setting were tuned manually through analysis of the corresponding loss curves.
Results can be seen in Figure 4. We can see that MNIST-like digits can be reconstructed even when inputs are from a different dataset. Class Mapping stitching tends to produce blurrier images. This is likely because, within each class pairing, images are matched randomly, which leads to an output that is essentially an average of all images in the class. Embedding Mapping stitching can produce cleaner MNIST images.
B.4. Stitching on Real-world Datasets
All stitches in Section are trained for 20 epochs using SGD with learning rate and weight decay of 10 momentum of 0.9, batch size of 128.
B.5. Training for Birdsong Recognition
We use publicly available code to train a ResNet-50 for birdsong recognition. We start with a random initialised ResNet-50 (unlike the original code) and use SGD to train for 20 epochs. We perform hyperparameter search over the learning rate
17

---

Functional Alignment Can Mislead: Examining Model Stitching
Table B.I. Results stitching two ResNet18 model together at different layers with different amounts of regularization Model was trained to predict the background colour on Colour-MNIST whereas model B was trained to predict the digit.
Sender Model
Receiver Model
Block
Block 2
Block 3
Block
Block 5
Linear
0.01
99.90%
99.96%
100.00%
100.00%
100.00%
100.00%
A
B
0.1
96.06%
99.09%
99.83%
59.96%
66.07%
99.92%
53.43%
22.23%
49.04%
96.66%
9.74%
9.74%
0.01
100.00%
100.00%
100.00%
100.00%
100.00%
100.00%
A
A
0.1
98.74%
93.12%
88.16%
85.74%
23.07%
88.97%
54.12%
38.46%
81.59%
86.72%
10.28%
10.28%
(10-4 to 10-1), weight decay (10-1 to 10-3) and momentum values (0.8 to 0.9). We believe that tuning the hyperparameters further would neither increase nor decrease the support for the point we are trying to make. If model stitching is a reliable model comparison method, should work for a range of models from high-performing ones all the way to models trained without hyperparameter tuning.
For training the final model the following configuration was used: learning rate of 10-2 weight decay of 10-4 and momentum of 0.9. Note that in the original manuscript we accidentally reported higher baseline for the birdsong model. However, in terms of layer stitch compatibility, all results remain the same (top-1 compatibility starts before the linear layer and top-5 compatibility starts before the 5th residual block).
B.6. Regularizing the Stitch
To explore different types of stitching, we experiment with training stitching with additional regularization. The first network A is ResNet18 trained on Colour-MNIST digits, but with the label corresponding to the background colour. The second network B, also a ResNet18, is trained on the same images, but now the labels correspond to the digit instead the background colour. We will stitch these networks together with a stitch S at layer such that our sender network and B our receiver. For the sake of comparison we will also look at stitching network into itself.
These stitches will be trained using BCE loss with additional L1 regularization term the parameters of the stitch. So our loss for this will be:
L = LBC + 1|00s11s
(I)
where LACE is our original loss function, parameters of the stitch. Results of additional LI regularization are shown Table B.1 where Block 3, Block 4, and so on refer to the layer at which the stitching took place. Percentages refer to accuracy of the final model. When our regularization strength À 0.01 we are still able to stitch A into B. This is not true for larger values of X, but larger values also prevents A being stitched into itself, suggesting that the regularization is too strong. In practice, see that even for À = 0.01, the regularization term still dominates the loss. This demonstrates that even with relatively strong LI regularization our stitching results are still valid.
scalar controlling the strength of our regularization, and Og are the
We repeat this experiment with weight decay and À both set to 0. Weight decay is L2 regularization, so this setting corresponds to training the stitch with no regularization at all. For this setting we achieve above 99% stitching accuracy for all layers, suggesting that our results are robust across different types of stitches, both with and without regularization.
C. Different Senders to Digit Receiver under Various Architectures
C.1. ResNet-18: 18: Stitching to Digit
As mentioned in Section 7 and Appendix B we used the Digit with Uncorrelated Colour (Digit) model as both receiver and sender in the stitch, and also used Clustered-Noiss as synthetic sender model. Here we present those extended results.
In Table C.1 we can see that the base accuracy of the Digit network is 0.978. When stitching Colour-Oni into this, accuracy improves to 0.999, indicating that Colour-Only is "better than" Digit. However, the base accuracy (against the same test set
18

---

Functional Alignment Can Mislead: Examining Model Stitching
Table C.I. Accuracy of trained Re models against 'Correlated' dataset (average of initialisations).
Acc. with stitch before Res Block 1
Model
Base Accuracy
'Digit' is Receiver
'Digit' Sender
Correlated
0.999
0.999
0.981
Digit
0.978
0.995
0.996
Colour
1.000
0.999
1.000
Colour-Oni -hlu
0.686
0.999
0.985
1.000
250
Stitched Model
Correlated
Test Accuracy
0.995
Feature Rank
200
Digit
0.990
150
Colour
Baseline
0.985
100
Colour- Only
0.980
50
Noise
Res Block 1
Res Block 2
Res Block
Res Block 4
Linear
Res Block 1 Res Block Res Block 3
Res Block
Linear
(a) Accuracy
(b) Rank
Figure C.I. (a) Test accuracy on Correlated data when the 'Noise', and each of the trained ResNet-18 18 models are stitched into the Digit receiver (baseline is the accuracy the original Digit model test Correlated data). This extends Figure by including Colour-Only and Clustered-Nois Noise. Shaded areas cover 100% of results to highlight the variability. (b) Rank analysis of the stitched models' representations. Shaded areas mark standard deviation.
of Correlated) of the Colour-Only network is 0.686, but when stitching Digit into it as sender, the accuracy improves to 0.985, indicating that Digit is "better than" Colour-Only. This contradiction constitutes a problem for the claim that model stitching can be used to ascertain relative quality. Note also that stitching the Digit model into itself produces a performance improvement from 0.978 to 0.995 which challenges the belief that the x convolution with batchnorms stitch (Bansal al., 2021) does add capacity.
In Figure C.2 illustrate the fact that for each initialisation the stitched Digit model outperforms the unstitched baseline. Shown on the plot are the highest baseline to the lowest stitch accuracy in one model. Baseline 0.985, Acc min 0.986. Also show is lowest accuracy of any stitch in any experiment from its baseline. Baseline = 0.978, Acc_min = 0.984. All stitched Correlated and Colour models perform above all baseline results.
C.2. VGG19: Stitching to Digit
In variant of the ResNet-18 experiment from 4.1 we stitched from differently biased VGG19 models into ones trained on Digit data, using Correlated data to train and test the stitch (repeated for 4 differently initialised models). This is indicative of comparing models which have learned different biases and are evaluated on biased data. As can be seen Figure C.3 most models show approximately equivalent, or improved performance at stitch points even though the information represented is different.
We believe that being unable, sometimes, to stitch differently biased models is irrelevant: the issue is that if it is sometimes possible to form such stitch then one cannot use stitch-compatibility to identify equivalent information.
C.3. LeNet-like Models: Stitching to Digit
During the rebuttal period we started extending the stitching experiments from Section 4.1 with LeNet-like architecture compatible with 28 x 28 input size. Note that we only train the stitch for 10 epochs more stitch compatibilities could be found. However, for the purpose of our paper it suffices to show that stitching compatibility exists. Results are presented in Table C.2
Note that during the rebuttal we also tried stitching randomly initialised senders in the LeNet-like case. For the setting we
19

---

Functional Alignment Can Mislead: Examining Model Stitching
1.000
Min Stitch Gain from Best Baseline
Test Accuracy
0.995
Min Stitch Gain for Worst Acc
0.990
0.985
0.980
Res Block 1 Res Block 2 Res Block 3 Res Block 4
Linear
Figure C.2. As in Figure 2b we show the same Correlated, Colour and Digit results. We mark the stitched performance compared with baseline as Stitch Gain (i.e. negative Stitch Penalty) for two cases.
1.000
Stitched Model
Correlated
Test Accuracy
0.995
Digit
Colour
0.990
Colour-Only
Mix
0.985
BW
0.980
features.2 2
features. .00
features.23 23
features.30 30
classifier.
Figure C.3. Accuracy variously trained VGG19 Sender models stitched into Digit model at different points the network using Correlated dataset. Dashed grey line shows baseline performance of whole Digit model against Correlated dataset. Shaded areas cover 100% of results to highlight the variability. Mix a dataset comprising examples from MNIST, Correlated, Colour-Only. BW greyscale-MNIST
20

---

Functional Alignment Can Mislead: Examining Model Stitching
Table C.2. Accuracy (%) when stitching the model trained purely colour patches (Colour) and the model trained correlated digit and colour (Correlated) with a receiver trained on grayscale MNIST (Digit). Stitching compatibility can be easily found when stitching at at least one layer.
Conv
Avg Pool
Conv2
Avg Pool
Reference
Colour to Digit
76.94
76.68 78.66
97.94
97.34 92.90
88.19
Correlated to Digit
79.11
93.24
88.19
Table D.I. Accuracy (%) of trained models against same dataset (average of 3 initialisations). Values shown aid reading of Figure D.1a
Model
Test Accuracy
'Correlated'
100
'Digit'
98
'Colour'
100
'Colour-Only'
100
ran, we found that the trained sender was successfully stitching, whereas the randomly initialised sender failed to stitch. Re-running the experiment with different seeds after the rebuttal period we found that there were cases where randomly reinitialising also led to successful stitch. See Appendix G.5 for discussion on stitching with randomly initialised senders.
D. Further Experiments with Clustered-Noise
D.1. VGG19 Results for Clustered-Noise Senders
There is a reasonable question about whether ResNet architectures may respond to model stitching in a particular way because of the skip connections. To address this, we extended the testing with Clustered- data to include VGG19 models. initialisations were run. 50 Epochs for model and stitch training: bateh_size=64, SGD, 1r=le-2, momentum=0.9, weight decay=le-4.
Each trained model cut and stitched at the following points. We decided take sample rather than testing every convolutional layer to reduce experimental duration (Figure D.
Whole Model (image data)
Cut before features.2 (Noise data)
Cut before features. 10
Cut before features.23
Cut before features.30
Cut before classifier.0
record the rank the activations for the Clustered-Notss test dataset at the first convolutional layer in the receiver model after the stitch (Figure D.
Figure D. and Table D.1 show that 'Digit' has improved results with stitching in the 'Clustered-Noise" dataset. 'Correlated' maintains accuracy. This is in accordance with the results for ResNet.
Most notably, 'Colour' and 'Colour-Only' show very varied results with stitching. It is not clear from this limited experiment whether that is due to the network not being manifestation the random variability shown by Csiszárik et al. (2021) (i.e. stitching from different stitch initialisations can yield very different stitching penalties). stichinng-pealttes)..
with the synthetic Clustered-Noise data, or if it is a
21

---

Functional Alignment Can Mislead: Examining Model Stitching
1.0
RCV Model
Test Accuracy
Feature Rank
300
Correlated
0.9
Digit
200
Colour
Colour-Only
0.8
Baseline
100
0.7
Whole
features.2
features.10 10
features.23
features.30 classifier.0
features.2
features.10
features.23
features.
classifier.O
(a) Accuracy
(b) Rank
Figure D.I. (a) Accuracy VGG19 models trained on the four dataset variants when Clustered-Noise stitched at each 5 different points in the model. The first sample shown ('Whole") is for the uncut models tested with their own test datasets. Shaded areas cover 100% of results to highlight the variability. Markers show individual measurements. (b) Ranks of activations just after the stitch from Clustered- test dataset. 'Baseline' shows ranks at the same points of the uncut 'Digit' trained VGG19 models when presented with 'Digit' test data. Shaded areas mark standard deviation.
We carried out small investigation using the model which performed worst at features. stitching. Trying multiple stitch initialisations gave accuracies in the range 40% - 80%. Trying multiple Clustered-Noiss dataset initialisations gave similar results. Lower and Higher learning rates were also tried with no change. It may be that the training parameters are wrong as sometimes good stitch-training loss increases between epochs, and we did not optimise stitch training hyperparameters. However, it may be that some models reach points in the loss landscape which it is hard to match when stitching from 'Noise' data.
This may represent difference between ResNet- 18 and VGG19 architectures. Two possible reasons (which deserve to be investigated) are:
Skip Connections ResNet skip connections may make stitching easier.
Dimensionality VGG19 at features.2 has 64 feature maps of size 32x32, whereas Res Block of ResNet-18 has 64 feature maps of size 7x7, The 1x1 Convolutional stitch layer can only learn create linear combinations of its input feature maps and it may be harder create necessary patterns of activation when (in VGG19) the maps have 21 times as many units.
The most obvious feature of the rank information in Figure D.lb is the difference between ranks in the unstitched 'Digit' network (presented with 'Digit' data) and the stitched networks presented with 'Noise' data. This demonstrates that being stitched a sender can produce similar functional results in terms of overall accuracy, while internal information different.
D.2. ResNet-50: Stitching 'Clustered-Noise" into ImageNet
can be argued that Colour MNIST-trained ResNet-18 models can be stitched from 'Noise' only because they are simple. To investigate this, we use ResNet-50 from torchvision pretrained on IMAGENETIK VI. The model is cut after the avgpool layer, before the final fully-connected linear layer, and prepended with the stitch create a stitch receiver model. The stitch (only) is then trained on the 'Noise' dataset, generated for 1000 classes. The stitch receiver model is then tested against the synthetic test dataset. This is repeated, cutting before other ResNet-50 Res Blocks (Table 1). At each Res Block from 3 onwards, accuracy exceeded the ImageNet baseline.
The experiment was repeated for some of the later layers for IMAGENETIK V2 (see Table D.2). Top-1 results only. Again, at these layers stitch performance exceeded the bascline (80.86%)
D.3. ResNet-18: Increasing the Cluster Size of "Clustered-Noise"
In the 'Noise' stitching experiments described above (Appendix B.2, D.I, D.2), the radius of each cluster was 0.1, resulting in tightly grouped class representations which are likely to be separable. We hypothesise that it is this separability which leads to high performance when stitching. To investigate, stitching was repeated with ResNet- 18 for one MNIST and one
22

---

Functional Alignment Can Mislead: Examining Model Stitching
Table D.2. Accuracy of ImageNet pretrained ResNet-50 models (IMAGENETIK, against 'Noise' dataset. Numbers of epochs adjusted to reduce processing time.
Cut before layer
Acc. (%)
Epochs
Res Block 3
99.8
3
Res Block 4
100
3
AvgPool
100
10
Linear
100
10
Table D.3. Varying the radius of noise clusters one stitch point for two trained models. Original model accuracy = MNIST: 98%,
Colour-Onie Only; 100%
Acc. (%)
Radius
MNIST
Colour-Only
0.5
99.7
99.8
0.9
97.3
97.8
0.99
96.4
96.3
1.0
96.0
96.6
Colour-Only model cut before Res Block for varying noise radii. For each model, the cluster centre positions were kept consistent while the data points around them were regenerated. Increasing the Clustered-Noiss radius decreased accuracy (see Table D.3) meaning that is possible to tune noise to have a stitching penalty above, below, or matching the original network. We argue that representational cluster shape may account for the quality of stitches between real-world networks.
D.4. Misaligned Class Size
During the rebuttal period we experimented with stitching smaller number of clusters (classes) into the pretrained Stylized and ImageNet receivers. We considered 10, and random noise clusters and successfully managed to stitch these into the receiver. As expected, the fewer classes represented, the easier was to achieve stitch compatibility. In the case of stitching two classes to the ImageNet model, for example, 100% accuracy was achieved within only one epoch of training.
E. Different Offsets for Remapping Classes
In subsection 5.3 we describe experiments in which a trained network cut and self-stitched; the original model's dataset used train the stitch, but with an offset in the class labels. We present and discuss further results here.
Performance sometimes reduced in earlier layers (see Figure E.1) for some datasets, but note that Colour-Only could sometimes achieve perfect performance even when stitching before Res Block 1.
BW (greyscale-MNIST) always suffered some degradation at early layers, but later in the network was successful. As short investigation, we ran the MNIST offset tests for single model at all 10 offsets and found similar curves for all offsets other than zero. We hypothesise that this is due to different classes having very varied clustering at early layers, but investigating this is beyond the scope of this paper.
F. Alternative Baselines
As described in section 5, we present stitching data with reference to both baselines in Table F.1. Note that in the main body of the paper we choose the most competitive between the two. While this is usually the performance of the receiver network on the receiver task, there exists one exception. Stylized performs better on ImageNet than on Stylized and therefore we choose the performance on sender's 's task as the baseline. Note that with the alternative baseline we observe full stitch compatibility.
23

---

Functional Alignment Can Mislead: Examining Model Stitching
10
1.0
Test Accuracy
Class offset -
Class offset
Correlated Colour-Only
Correlated
Correlated
X
Colour-Only
Colour-Only
BW
0.8
0.8
0.8
Res Block
Res Block
Res Biock
Res Block
Linear
Res Bleck
Res Block
Res Block
Res Block
Linear
Res Bleck
Res Block
Res Block
Res Block
Lineer
(a) Offiet 1
(b) Offset 2
(c) Offset
Figure E.I. accuracy shown three initialisations of three datasets stitching before each layers for different class offsets. Shaded regions cover 100% of results to highlight variability
Table F.I. Results of stitching various models into and from ImageNet. We highlight in in bold the representations that were stiich-compatibl.. Note that "Block X" and "Lincar" means that we are stitching before the Xth residual block and the classification layer respectively. Sender Baseline is the performance of the receiver the given task (sender's task), while Receiver Baseline is the performance of the receiver on the receiver task. *These are the accuracy measurements we obtain when resizing the test samples to 256, as described in the paper. We only match (Geirhos et al., 2019)'s reported top-5 accuracy of 16.4% (with corresponding 73.8% top-1 accuracy) when removing the resize operation.
Sender Baseline
Receiver Baseline
Block
Block
Block
Block
Block
Linear
ImageNet to Stylized
Top-1 Top-5
58.39
58.61
61.53
65.31
70.43
74.34
60.18
56.18
80.57
80.61
83.28
86.13
89.78
91.93
82.60
78.96
Stylized to ImageNet
Top-1 Top-5
10.61
11.23
17.60
30.26
45.45
49.17
7.15*
76.13
22.65
23.08
34.34
52.52
69.57
72.30
15.87*
92.86
10-class ImageNet to Birdsong
Top-1 Top-5
15.40
19.00
25.00
46.20
75.60
88.40
6.59
77.98
63,80
68.40
76.40
88.60
98.60
99.80
50.40
96.07
Clustered-Noise to ImageNet
Top-1 4.77
4.11
96.59
100
100
100
76.13
Top-5
13.51
11.99
99.77
100
100
100
92.86
G. Comparison with Randomly Initialised Sender Networks
As part of the evaluation of the potency of the stitching architecture, Bansal et al. (2021) examined the performance when stitching from a randomly initialised sender network, showing that stitching penalty increases (accuracy decreases) as one stitches at later points in the networks.
Importantly, note that this not given as a precondition for the application of stitching to model comparisons. Bansal al. (2021) simply perform this experiment for one of the many architectures they consider way of confirming that the stitching layer "is not performing learning". Nonetheless, experiments with discussion (Appendix G.5) on how they align with the perspective presented in this paper.
examined it in some of our settings. We conclude this series of
G.1. ResNet-18: Randomly Initialised Sender to Digit
We extend the setting of Figure C.1 by attempting to stitch randomly initialised (untrained) networks into Digit receiver using Correlated dataset. This was repeated for the same receiver but using different seed for the random network and stitch process (see Figure G.1).
As was seen in Figure C. la, stitching improves performance relative to the baseline. The randomly initialised models perform better than the Digit model, but less well than Correlated, Colour, Colour-Only and Clustered-Noise.
G.2. ResNet- 18: Randomly Initialised Sender to BW Receiver with Colour-Only Dataset (and to Colour-Only Receiver with BW Dataset)
In subsection 4.2 we indicated that using Colour-Only senders and stitch training datasets with Greyscale receivers results in 100% accuracy. For comparison, we tested for a single Greyscale-MNiN MNIST receiver stitching randomly initialised
24

---

Functional Alignment Can Mislead: Examining Model Stitching
0.995
Stitched Model
Digit
Test Accuracy
Randomly Initialised
0.990
Baseline
0.985
0.980
Layer
Layer
Layer
Layer
Linear
Figure G.I. ResNet-18 trained Digit compared with stitching from two randomly initialised sender networks. Stitch training and evaluation used Correlated dataset. Digit plots are for self-stitching. Baseline whole Digit model tested with Correlated dataset
1.0
0.9
Test Accuracy
0.8
Stitched Model
0.7
Randomly Initialised
Colour-Only
0.6
BW
x
Layer 1
Layer 2
Layer 3
Layer 4
Linear
Figure G.2. ResNet-18 trained on Colour-Only stitching from randomly initialised sender network compared with sender trained on BW data. Stitch training and evaluation used Creyscale-MNIST (BW) dataset. Also shown is self-stitching with Colour- Only as its sender, also on BW dataset.
sender with Colour-Only data and found that it also achieved 100% accuracy at each layer. The same result was obtained self-stitching the same Greyscale-MND MNIST model as its own sender.
We also did the complementary experiment Colour- Colour-Only receiver, and Greyscale-MNIST datasets for the stitching (Fig- ure G.2). The Randomly initialised model underperforms relative to the BW sender model expected, given that the BW-trained sender model should be able to separate BW data. It also outperforms the Colour-Onl Onlur--Oly sender stitched into itself as receiver.
G.3. VGG19; Randomly Initialised Sender to Digit with Digit Dataset
For arbitrarily selected VGG19 model trained on Digit dataset, found baseline performance very similar to stitching the network to itself at each layer (Figure G.3). By comparison, stitching with Digit dataset from uninitialised sender shows performance degrades with the depth of stitch expected.
Varying the experiment to more closely mirror the ResNet-18 work in Figure G. and the VGG19 work in Figure C.3, we find that training the Digit stitch on the Correlated dataset gives an improvement in performance relative to the baseline of Digit model tested on Correlated dataset. However, this is outperformed by the uninitialised sender (see Figure G.4).
G.4. Linear Probing Randomly Initialised Senders
To give more perspective on the results obtained in this section, we carry out an additional set of experiments where we verify the level of linear separability of representations obtained by passing datasets through randomly initialised senders.
25

---

Functional Alignment Can Mislead: Examining Model Stitching
0.98
Test Accuracy
0.96
0.94
Stitched Model
Digit
0.92
Randomly Initialised
X
0.90
baseline
features.2
features. 10
features.23
features.30
classifier.0
Figure G.3. VGG19 model (trained Digit) stitched itself (using Digit datasct) compared with stitching in uninitialised network using the same dataset. Baseline is whole Digit model on Digit dataset.
1.000
x
X
Stitched Model
Test Accuracy
0.995
Digit
Randomly Initialised
baseline
0.990
0.985
features.2 2
features.10 10
features.23
features.30
classifier.0 0
Figure G.4. VGG19 model (trained on Digit) stitched to itself (using Correlated dataset) compared with stitching in an uninitialised network using the same dataset. Baseline is whole Digit model tested with Correlated dataset.
26

---

Functional Alignment Can Mislead: Examining Model Stitching
Conv 1
Avg Pool 1
Conv2
Avg Pool 2
Colour- Only
25.33-6.00
98. 98.88 10.76
26.94_2.68
88.2049.07
Greyscale MNIST
95.12
92.78
86.15 86.1540.91 20.04_0.43
Table G.I. Test accuracy when probing untrained LeNet-like senders for Colour-Only and Grayscale MNIST data.
Block 1
Block 2
Block 3
Block 4
Block 5
Linear
Colour-Only
10.00
=00 28.33
10.00 98.2640.11
=0000
100+0.00 93.77
100+0.00
Greyscale MNIST
97. 10.07
97.01 20.21
82.88
Table G.2. Test accuracy when probing untrained ResNet- senders for Colour- Onll and Grayscale MNIST data.
To do so, we linearly probe these representations. That is, we put the representations through a linear layer and attempt to classify these. We train the probe (linear layer) for 20 epochs, using SGD with a momentum of 0.9 and weight decay of 10-4 learning rate of 10-3, batch size of 128. We perform 5 runs and report the mean and standard deviation.
Note that did not perform any hyperparameter tuning and it is possible that training for longer with more carefully chosen hyperparameters, a higher degree of linear separability might be found. We simply want to show that good separability can be found. We perform these experiments for the Colour-Only Onll and Greyscale MNIST datasets. The results for LeNet-like untrained senders are given Table G.I, while those for untrained ResNet- 18 senders can be found in Table G.2.
G.S. Discussion on Randomly Initialised Senders
In this section we have seen that there are cases in which untrained versions of the models we considered can be successfully stitched (and also cases in which this was not achieved with the standard hyperparameters and training budget.) Are our results diminished by cases where stitching from an untrained model was possible? We believe they further amplify the central message of our paper.
Consider the case of putting Colour-Only data through untrained convolutions. As have seen, there many cases where colour information leaks through untrained convolutions, and representations remain clustered (see Appendix G.4 where we verify this in couple of cases with a linear probe). In the light of the experiments in Section 5, we argue that stitch compatibility of untrained senders can be understood from the perspective of clustering. As long as a mapping can be found, we can abstract away from the information that the representational clusters were originally based on. Whether is birdsongs, natural images, stylised images, random clustered noise, or even colour patches passed through untrained convolutions, the stitch need only find mapping to the receiver. The stitch and receiver become a complex but fixed probe.
27


Reviews:
[{"review_id": "ChoNV7tfAD", "reviewer": "Reviewer_HH2d", "summary": "This paper contributes to the representation alignment field. The core message is that the functional similarity of representation spaces, measured by stitching performance, is not correlated with information content. This is shown empirically through well-controlled settings (where the variation factors are known) in classification and autoencoding tasks. A final discussion is provided on the implications of this finding for the representation alignment research.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the useful recommendations, questions, and very constructive review.\n\n\n## Experimental Design or Analyses\n\n> Even after reading the corresponding section in the supplementary material, I had trouble understanding the \"Embedding Mapping\" procedure for AE stitching (Section 6). My understanding is that, in this case, the stitching layer is optimized to map a sample from model A's encoding space to model B's encoding space, drawing samples from the joint dataset and not from any model-specific one. Can the authors kindly confirm/clarify this?\n\nThank you for pointing out the missing clarification here. We believe the stated understanding is correct. It can effectively be considered that we create a joint dataset where we pair up each training sample from dataset A (used to train encoder A) with another training sample from dataset B (used to train encoder B). Training samples are paired up using the encodings (i.e. in the embedding space).  Specifically, we solve the linear sum assignment problem in the AEs bottleneck to pair up samples from datasets A and B. Passing an image from dataset A through encoder A, the stitch is trained to map it to its corresponding sample from dataset B, passed through encoder B. Does this help clarify our approach? \n\n## Essential References Not Discussed\nThank you for pointing out the miscitation on L071 and for the suggested additional references, which we have now included.\n\n## Weaknesses\n> The experiments are relatively small-scale regarding the number of classes, datasets, and models used. Scaling these aspects up would further solidify the findings and make them more persuasive. That said, this does not diminish the significance of the results.\n\nWe agree that including more models and data sets would further strengthen the point that model stitching is not adequately capturing information similarity in an even broader variety of settings. We also agree with you that this does not diminish the significance of our results since our contention is that it is sufficient to show that model stitching can be deceived into declaring incorrect matches in many cases (spurious correlations, different data sets, different modalities, etc.). These cases involve both carefully constructed artificial data sets as well as a number of real-world data sets including ImageNet. Note that to address the rebuttals, we carried out a number of additional experiments. These include a LeNet-like architecture for a number of our core experiments (requested by reviewer rqks) or stitching between non-matching number of classes (reviewer 93zT). We are happy to continue considering additional settings. Is there one particular data set or architecture that would significantly strengthen our paper? We will do our best to include these, provided that they are within our computational budget.\n\n>The final paragraph of Section 7 (“Are our results possible because models reached a shared understanding of reality?”) could be reworded, specifying \"functionally aligned\"\n\nWe are happy to clarify that it is functional alignment that this paper targets (which Huh et al. use as supporting evidence, although their proposed method is indeed structural). Just to confirm, do you mean L369--381? \n\n\n## Other comments or suggestions:\n\n* Thank you for the suggestion to emphasise the autoencoding experiments. We will address this in the revised manuscript. Would adding the following sentence to the abstract suffice “(...) We then show that clustered random noise, and models trained to solve entirely different tasks on different data modalities, can be successfully stitched into MNIST or ImageNet-trained models. _Even autoencoders trained on different data sets can be connected to each other._ (...)”?\n\n* “CIFAR-10 is mentioned abruptly”: Thank you for pointing this out. We will change this explanation to refer to Fashion-MNIST and MNIST, in line with the previous paragraph. We also added the missing CIFAR-10 citation. \n\nWe would like to thank you once again for the positive review and suggested improvements. If we omitted something in our response or if our clarification did not answer your questions, we would kindly ask the reviewer to let us know.", "claims_and_evidence": "The claims made in this paper are exceptionally well-supported by empirical evidence. The experiments are not only well-designed but also structured incrementally, progressively introducing additional variation factors to strengthen the validity of the findings. Moreover, the results are clearly presented and effectively communicated.", "methods_and_evaluation_criteria": "This paper has no proposed \"methods\"; it revolves around analyzing the relationship between functional similarity and \"information content\". Therefore, model stitching performance is the evaluation itself. It is applied in the standard stitching procedure, with a (much appreciated) change in the reported baseline, stronger than the one commonly used in the literature.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Experiments are particularly in line with the existing literature and look very solid, with clear experimental settings and accompanying results.\n\nEven after reading the corresponding section in the supplementary material, I had trouble understanding the \"Embedding Mapping\" procedure for AE stitching (Section 6). My understanding is that, in this case, the stitching layer is optimized to map a sample from model A's encoding space to model B's encoding space, drawing samples from the joint dataset and not from any model-specific one. Can the authors kindly confirm/clarify this?", "relation_to_broader_scientific_literature": "The paper is well situated in the representation alignment literature, challenging the common belief that functional similarity is a good proxy for information content. I would go as far as saying that this paper is a must-read for anyone working in the field of representation alignment, especially as its findings and discussions imply a reconsideration of current practices and encourage more careful framing of results in future studies.", "essential_references_not_discussed": "I think references are generally well chosen and discussed, with a couple of exceptions:\n- Section 1, line 071. Why is Bansal et al. 2021 cited for model stitching and not the (later correctly referenced) Lenc & Vedaldi 2015? Did the authors intend to target the \"good networks learn similar representations\" aspect?\n- A couple of relevant works on representation alignment methods with stitching applications are missing: \n  - [a] Proposes reusable components for stitching purposes, a missing reference for the stitching ones.\n  - [b] Analyzes the zero-shot compatibility (mostly through stitching) of networks under various variation factors.\n\n[a] Towards Reusable Network Components by Learning Compatible Representations; Michael Gygli, Jasper Uijlings, Vittorio Ferrari; AAAI 2021;\n[b] Relative representations enable zero-shot latent space communication; Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, Emanuele Rodolà; ICLR 2023;", "other_strengths_and_weaknesses": "### **Weaknesses**\n- The experiments are relatively small-scale regarding the number of classes, datasets, and models used. Scaling these aspects up would further solidify the findings and make them more persuasive. That said, this does not diminish the significance of the results.\n\n- The final paragraph of Section 7 (“Are our results possible because models reached a shared understanding of reality?”) could be reworded, specifying \"functionally aligned\". The Relative Representations paper (Moschella et al.) and The Platonic Representation Hypothesis (Huh et al.) both suggest that the structure of representation spaces is shared across models. In Moschella et al., it is shown that point-wise distances are similar across spaces, and in Huh et al., this compatibility analysis is scaled up in terms of models considered, but again, a similar structural metric is used (Jaccard similarity between sample neighbors across spaces), not a functional one. I would argue that a high structural alignment can, in fact, be seen as a somewhat shared conceptual representation.\n\n### **Strengths**\n\nI commend the authors for the clarity and logical flow of the paper and the incremental complexity of the experiments, which strengthens the validity of the claims.", "other_comments_or_suggestions": "- Personally, I would give more prominence to the autoencoding experiments, as they are among the most interesting ones, and they are not directly mentioned in the abstract/conclusions. The classification setting can be intuitively explained by decision boundaries accommodating linearly transferred samples, but the autoencoding case presents a more complex and intriguing scenario.\n- In section 6, Fumero et al. is cited as \"extending the definition of model stitching to the generative case\". Stitching in autoencoders was previously shown in **[b]** (Moschella et al., 2023)\n- In section 6.6, Class Mapping paragraph: CIFAR-10 is mentioned abruptly;\n- In \"Should stitching then be used as a measure of model quality?\" (Section 7), I would suggest adding a reference to **[c]** at the end of the first paragraph, as this work directly ties the linear separability of the representations to the representational capacity.\n\n**[c]** Towards an Improved Understanding and Utilization of Maximum Manifold Capacity Representations; Rylan Schaeffer, Victor Lecomte, Dhruv Bhandarkar Pai, Andres Carranza, Berivan Isik, Alyssa Unell, Mikail Khona, Thomas Yerxa, Yann LeCun, SueYeon Chung, Andrey Gromov, Ravid Shwartz-Ziv, Sanmi Koyejo; ArXiv 2024;", "questions_for_authors": "N/A (already addressed in the other sections).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pyzeHvwYeh", "reviewer": "Reviewer_WvUr", "summary": "This paper suggests rethinking the use of model stitching as a representation comparison tool. Model stitching is a functional approach to comparing representations of two models. Essentially one glues the first $k$ layers of one model with the last $\\ell$ layers of another model with 1-2 *stitching* layers. The stitching layers are finetuned for the task while the components of the original models remain frozen. The idea is that if one model can use another’s activations to solve a task (with minor translation from the stitiching layers), then they must have had similar representations. The paper’s main point is that model stitching can show similarity in many cases where we would not expect similar representations and thus we should be careful about how we interpret model stitching results. Examples in the paper include: (i) models that have learned very different shortcuts on a dataset (color vs shape bias) can be effectively stitched together, (ii) models that have been trained to identify random noise can be stitched to a model trained on image data, and (iii) models trained on distinct modalities can be stitched together (images vs spectrograms). Finally, the paper argues that these results suggest fundamental limitations of stitching rather than other possibilities (such as convergent representations).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the useful suggestions and detailed review.\n\n## Claims and Evidence\n\nWe agree that the paper can stand on its own without the broader statements about the functional perspective. We will restrict these to the discussion section. Would this fully address the concern?\n\n## Methods and Evaluation\n\nTo address the point about different types of stitching we considered training the stitch with various degrees of L1 and L2 regularisation. For all the experiments we reproduced with the added regularisation term, we found that with mild regularisation, we are still able to stitch successfully. For more aggressive regularisation, we fail to stitch. To verify that this is due to learnability issues, we aim to stitch the same model to itself. That is, we break the same model instance up into a sender and a receiver and we try to stitch back up the model to itself (we know that there exists at least one mapping in this case - identity). This led to a stitching failure, indicating that for aggressive regularisation there is an issue of learnability that stops models from being stitched together successfully. In all the models where a model was successfully stitched to itself, we were also able to stitch between networks that learned different information. Therefore, our observations remain valid for different stitching options.\n\n## Experimental Design or Analyses\n\nThe numerical rank of representations provides a crude estimation of their compression and was used in recent publications [e.g.  1, 2]. We agree with the reviewer that the rank does not take into account which of the feature maps meaningfully contribute to the model’s output (i.e. are salient). However, we believe the alternative statistics proposed in the review suffer from the same limitation. We are unaware of any computationally feasible techniques which are reliably capturing this for the types of models we consider.\n\nWe also agree that alternative invariants could be computed. However, we would equally struggle to motivate any particular one, since each has its limitations. If the reviewer believes there is a particular one which the paper would benefit from including, we are happy to add it in the revised manuscript. Note that our objective was simply to include additional evidence that the sender representations are not entirely equivalent when processed by the receiver (even if they only differ in meaningless dimensions). However, this point is far outweighed by our subsequent experiments and therefore the rank experiments could have been omitted altogether without affecting the contributions of the paper. \n\n[1] Masarczyk, W., Ostaszewski, M., Imani, E., Pascanu, R., Miłoś, P. and Trzcinski, T., 2023. The tunnel effect: Building data representations in deep neural networks. Advances in Neural Information Processing Systems, 36, pp.76772-76805.\n\n[2] Feng, R., Zheng, K., Huang, Y., Zhao, D., Jordan, M. and Zha, Z.J., 2022. Rank diminishing in deep neural networks. Advances in Neural Information Processing Systems, 35, pp.33054-33065.\n\n## Relation to Broader Scientific Literature\nThank you for the suggestion to situate our work within the broader representation comparison literature beyond functional methods and XAI more broadly. We are happy to include this in the manuscript.\n\n## Other Strengths and Weaknesses\n\nWe agree that the real-world experiments could be found more convincing than the artificial ones. However, as noted by reviewer HH2d, these artificial experiments allowed us to incrementally construct our argument, starting with settings where we had full control over the data and therefore could rule out alternative interpretations of our results. We treat the experiments on real-world data as additional evidence to our core experiments on spurious correlations, since in the real-world case we do not have a ground truth, and alternative explanations could be imagined. As far as we understand, reviewer rqks pointed out exactly these possible alternative explanations. Therefore, without controlled, incremental experiments, we would not be able to reliably show stitching’s limitations. We also believe that it is in the contexts where similar data is used that researchers are most likely to expect (and want to test for) the representation of similar information in their networks: therefore the core experiments are most likely to be persuasive and relevant to the intended audience.\n\n## Other comments and suggestions\n\nDid the reviewer mean Figure 1 instead of 3? If so, we will adjust the saturation.\n\nWe once again thank the reviewer for their time and would kindly ask them to let us know if we did not address the raised concerns.", "claims_and_evidence": "Overall, the reviewer believes that most of the core claims related to stitching are well-supported. It is where the paper makes claims about the functional perspective more generally that issues arise. For instance:\n\n- Line 069: “We argue that in this context, the functional perspective alone does not lead to a meaningful comparison.” While the paper does a very good job providing support for the idea that stitching (in its usual form), is not reliable in many settings (including spurious correlations to which this quote refers), it does not provide much evidence that the functional perspective does not lead to meaningful comparison. This is a far stronger statement and would require an argument that takes into account all possible functional approaches to representation comparison (including those that don’t exist yet).\n- Line 090: This issue where results about stitching are used as evidence against the functional approach generally appears again in a statement about functional approaches and model compatibility.\n\nThe reviewer believes that paper can stand on its own without these broader statements and would be improved if they were either softened or removed.", "methods_and_evaluation_criteria": "The reviewer was satisfied with the evaluation. The paper describes experiments in a satisfactory number of settings, including: (i) models learning different shortcuts, (ii) models learning on different datasets, and (iii) models learning on different modalities. \n\nA deeper study of stitching hyperparameters, including different types of stitching layers, would help understand where and how stitching is failing to capture meaningful signal.", "theoretical_claims": "The paper did not make any theoretical claims.", "experimental_designs_or_analyses": "Overall, the reviewer found the experimental design and analysis to be thorough and robust. There were a few small issues that stood out.\n\n**Numerical rank:** The reviewer felt that the use of numerical rank as an invariant of representations could have been better motivated. There are a number of different invariants that one can calculate for representations (e.g., various geometric, topological, information-theoretic statistics). Numerical rank is a reasonable choice but certainly not the only one and some argument should be provided for why this was chosen among others. For instance, it may be that high-rank and low-rank representations only differ in relatively meaningless dimensions? That is, taking a high-rank representation and replacing it with a low-rank approximation would not actually change the behavior of the model?", "relation_to_broader_scientific_literature": "The paper does a good job summarizing past work on stitching but the reviewer believes that it would be helpful to also situate the work within the line of research exploring the limitations of representation comparison methods and XAI methods in general. For instance [1] or [2]. \n\nThe reviewer believes that this work is a strong contribution to that tradition.\n\n[1] Davari, MohammadReza, et al. \"Reliability of cka as a similarity measure in deep learning.\" arXiv preprint arXiv:2210.16156 (2022).\n\n[2] Ding, Frances, Jean-Stanislas Denain, and Jacob Steinhardt. \"Grounding representation similarity through statistical testing.\" Advances in Neural Information Processing Systems 34 (2021): 1556-1568.", "essential_references_not_discussed": "They are not essential but there are analogous works that have been performed for other types of representation similarity or XAI methods. It would be good to include some of these.", "other_strengths_and_weaknesses": "**Strengths:** \n\n- **Clarity:** Overall, the paper is clearly written. The paper makes its arguments directly and concisely. The reviewer found this easy to read.\n\n**Weaknesses:**\n\n- **The issue of spurious correlations:** The reviewer found the experiments on spurious correlations interesting, but also worried that they are somewhat overshadowed by subsequent experiments which are much more dramatic. Stitching not catching whether two models use different shortcuts for a given task pales in comparison to stitching not catching the difference between two models trained for completely different modalities/tasks.", "other_comments_or_suggestions": "- The images in Figure three (3rd and 4th rows) have very odd colors. Are they supposed to look psychedelic?", "questions_for_authors": "My questions are implicit in the comments above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q9OMysacR8", "reviewer": "Reviewer_93zT", "summary": "This work conducts an empirical evaluation of when functional alignment is not an indicator of the semantic similarity of the learned features. Functional alignment between models $A$ and $B$ is measured by the performance of a “stitched” model where an affine layer connects the first $l$ layers of $A$ and the last $l$ layers of $B$ (where $A$ is  the “sender” and $B$ is the “receiver”). Experiments in a controlled setting are conducted, where the stitched model is evaluated on a colored MNIST dataset for which both digit and background color correlate with the target label. When the sender and receiver models are trained to represent different semantic features, the stitched model achieves improved performance, demonstrating that functional alignment can be high in spite of the models learning different semantics. Additional results on autoencoders and real world datasets support the claim that model stitching performance is not necessarily indicative of information similarity.\n\n## update after rebuttal\nI will keep my current score, as my clarification questions have been addressed.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and for the positive comments.\n\n## Experimental Design or Analyses:\n\n> One potential concern is that no hyperparameter tuning was done for the sender or receiver models on the real-world data (although many experiments use pretrained ImageNet models).\n\nTo address this, we started experimenting with a few different learning rates and momentum values. So far we have not noted any significant improvement in performance but we are happy to try additional configurations if the reviewer believes this would impact the quality of the paper. However, we believe that tuning the hyperparameters would neither increase nor decrease the support for the point we are trying to make. If model stitching is a reliable model comparison method, it should work for both high-performing models as well as models trained without hyperparameter tuning.\n\n## Other Strengths and Weaknesses\n\n* W1: See above our answer to the Experimental Design and Analyses section.\n* W2: We ran the experiment suggested by the reviewer. We modified the noise data to represent only a small number of classes and mapped it to an ImageNet-trained receiver (1000 classes). We experimented with 10, 5, and 2 classes for the noise data. We achieved full stitching compatibility in all these cases. This is not surprising, since the stitch only needs to be able to map the representations of the 10 clusters to any subset of size 10 of the 1000 ImageNet classes. As long as this mapping can be learned, the remaining 990 classes can be ignored. Did we correctly understand the experiment suggested by the reviewer?\n\n## Questions for Authors\n\n* Q1: We would like to thank the reviewer for pointing out this missing detail. For the autoencoder experiments had already tuned the learning rate, but omitted to state this in the manuscript. It is possible we may get better performance by further tuning the hyperparameters. However, the current setup is sufficient to demonstrate that stitching **can** align seemingly disparate representations in the unsupervised/generative setting. We are happy to consider further hyperparameter tuning if the reviewer believes this would significantly improve our paper, but we believe the same argument as in the discriminative case applies.\n\n* Q2: See W2.\n\nWe would kindly ask the reviewer to let us know if there are any concerns we did not address to a satisfactory level.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A.", "experimental_designs_or_analyses": "Yes, I checked the stitching experiments and looked through the Appendix B. One potential concern is that no hyperparameter tuning was done for the sender or receiver models on the real-world data (although many experiments use pretrained ImageNet models).", "relation_to_broader_scientific_literature": "Model stitching has been studied in the past [1, 2] as a way to measure the representation similarity between neural networks trained with different settings (e.g.  initializations, different datasets). These works find that better performing networks have higher functional alignment and assume that the reason for the high alignment is due to different representations capturing similar information. This paper probes whether this intuition holds by controlling for the information that the networks can learn through varying the training data. \n\n[1] Bansal et al. “Revisiting Model Stitching to Compare Neural Representations” (2020).\n\n[2] Csiszarik et al. “Similarity and Matching of Neural Network Representations” (2021).", "essential_references_not_discussed": "None that I know of.", "other_strengths_and_weaknesses": "Strengths\n1. The problem of understanding whether model stitching is misleading is interesting and well-motivated.\n2. Experiments in the controlled colored MNIST setting support the claim that high stitching performance can be achieved without models representing similar information. \n3. Experiments outside of the colored MNIST setting are comprehensive, as the authors experiment with stitching cross-modal representations and autoencoders. \n\nWeaknesses\n1. It doesn’t seem like hyperparameters for the sender or receiver models were tuned, which may be a concern for real world data (particularly when the accuracy is not close to 100%).\n2. In Section 7, the authors present a potential hypothesis that high functional alignment is indicative of the clustering quality of the sender’s representations. However this hypothesis is not fully explored in the sense that the number of clusters (classes) that the sender is trained on is the same as the number that the receiver is expecting. It would be interesting to see how the stitching performance varies when the number of clusters is misaligned.", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "1. Would tuning hyperparameters in the autoencoder setting improve the performance of the class reconstruction method?\n2. How does the stitching performance affected when the number of clusters that the sender and receiver models are trained to represent are not the same? For example, if the number of clusters in the noisy data is less than 10, would stitching always yield worse performance? \n\nWhile I believe these experiments would strengthen the paper, I think that the evidence in the paper that functional alignment is misleading is compelling and well-supported.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4npmuaLf5f", "reviewer": "Reviewer_rqks", "summary": "The authors investigate the suitability of model stitching as a tool for analyzing the informational content of feature representations. Specifically, the authors show that models trained for different tasks (which arguably encode different information in their representations) could be stitched together to achieve high accuracy on a corresponding dataset. The authors demonstrate this behavior in several settings: in a controlled experiment using the Colored MNIST task, in a more realistic setting using ImageNet-type datasets, and in an autoencoding task. These results indicate that the model stitching does not identify crucial differences between feature representations, which limits its applicability for representational comparison.\n\n## update after rebuttal\n\nI am inclined to keep my current score. As I argued in my Rebuttal Comment, I think the current methodology generally does not test the model stitching as the measure of the informational content of the **initial representation function**, and instead tests the similarity of the **final classification functions**. Thus, I still think that the current version is **misleading**.\n\nSpecifically, the experiments in Section 4 indeed test the inapplicability of model stitching as the measure of **representation function** similarity. However, the message of this section is weaker than the message of the paper: it only identifies the **failure mode** of model stitching in the situation **where the stitching dataset has two distinct patterns that perfectly explain the data labelling**.\n\nAs for Section 5, I still insist that these experiments are different in scope and test only the applicability of model stitching for the comparison of **final classification functions**, and not the **initial representation functions**. I think this result is weaker and should be positioned differently.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for stating their understanding of the paper and for the time they dedicated to structure the evaluation of our paper.\n\n## Questions For Authors\n\n* Q1: The reviewer has misunderstood our methodology. While point 1 is correct, points 2--5 are incorrect. \n    * 2&3: we are training and evaluating the stitch on dataset $D_{A}$.\n    * 4: We are not stitching a model that was **trained** on clustered noise to a receiver. We remove the sender and supply the stitch with clustered random noise matching the dimension of the expected feature maps.\n    * 5: Since there is no sender, the stitched model is evaluated on newly sampled random noise using the same cluster centres as in the training set.\n* Q2: 100% accuracy was achieved on clustered noise, not on the ImageNet data set.\n* Q3: Does the reviewer mean L417 “While there might be cases in which stitching is insightful”? All we wanted to say is that we don’t exclude the possibility that applications could be found. We did consider several possible interpretations of model stitching, but did not find stitching to be insightful. In L421-427 we discuss why we believe other methods are more suitable.\n* Q4: The reviewer suggested two alternatives for stitching: 1. Rescaling of channels: Due to network symmetries this would not reliably work even if two networks were capturing the same information. 2. Regularisation: to address this we experimented with L1 of varying strength and our claims remain valid. Please see our response to reviewer WvUr for details.\n* Q5: We have re-run some of our spurious correlation experiments with a LeNet implementation compatible with a 28x28 input size. Our findings remain valid for this network. For LeNet we also tried stitching to a **randomly initialised** sender as suggested by the reviewer and we can confirm that stitching fails in this case, further strengthening our results.\n* Q6: Our paper shows stitching's inability to identify dissimilarity between networks. It is outside the scope to propose **model quality** tools. Nonetheless, we agree that the community should aim to define additional notions of model quality beyond performance on held-out data, robustness, calibration, representation clustering, etc.\n\n## Other Comments or Suggestions\nWe thank the reviewer for pointing this ambiguity. We had stitched the ImageNet sender to a Birdsong receiver. We have now clarified this in the manuscript.\n\n## Other Strengths and Weaknesses\n\n* W1: Thank you. We have now fixed this.\n* W2: Thank you. We created a diagram further explaining the methodology (including data sets used for training and evaluating the stitches) and will include it in the revised paper.\n* W3: A standard AE trained on MNIST doesn't reliably generate meaningful images, particularly from arbitrary latent space samples (a key VAE motivation). Please let us know if the interpretation of these results in L369–373 needs extending. The relevance to the community was noted by reviewers HH2d and 93zT.\n\n## Supplementary Material\n\nThe contradiction stems from the reviewer’s misunderstanding (Q1). Thank you for highlighting the missing bit of code. Upon publication the full repository will be publicly released.\n\n## Methods and Evaluation Criteria\n\nFor LeNet, see Q5. We did not tune hyperparameters as it suffices to show that one map between different representations exists. We believe showing that we can stitch even without tuning hyperparameters makes a more compelling case.\n\n## Claims and Evidence\n\n* E1: See Q1.4 and Q2.\n* E2: This alternative explanation motivated our core experiments on artificial data sets. There, we know exactly what patterns exist in the data refuting the alternative. Experiments on real-world data complement them. In L253--260, we acknowledge that in the absence of ground truth, alternative interpretations exist.\n* E3: Note that Bansal et al. (2021) do not introduce the performance of the randomly initialised network as a precondition for the applicability of the model stitching to a specific context. Rather, they use it as evidence that their proposed stitching methodology works. Nonetheless, we have now performed this reference experiment on a few models. For the considered experiments we find that randomly initialised networks failed to stitch, yet our models were successfully stitched together. Therefore our observations hold even with this additional requirement (although it was not proposed as a requirement by Bansal et al. (2021)).\n* E4: See Q4 and E3.\n* E5: Could the reviewer let us know if they have any remaining concerns and how they would need these to be addressed? We would like to reiterate that the point we are trying to make is that while model stitching **may** correctly identify equivalent networks as **informationally equivalent**, we provided multiple counterexamples in which stitching incorrectly identifies as equivalent networks that process semantically distinct types of input data.", "claims_and_evidence": "I have some problems with comprehending the paper's explanations. So, I start with an outline of my understanding of the paper's methodology to clarify my evaluation.\n1. As I understand, when model $A$ trained on dataset $D_A$ (sender) is stitched to model $B$ trained on dataset $D_B$ (receiver), we get a new model $C$ that starts (from input) with model $A$'s layers and ends with model $B$'s layers.\n2. By default, the stitching layer of model $C$ is trained on the training portion of dataset $D_B$.\n3. By default, model $C$ is evaluated on the test portion of dataset $D_B$.\n4. By stitching \"clustered noise\" to models, the authors mean stitching models trained on clustered noise to models.\n5. All stitches with models trained on clustered noise were evaluated on the receiver's dataset.\n\nNow, I describe my evaluation of the paper's claims.\n1. Given my understanding, the results in Table 1 can not be correct. Specifically, in the last row, the stitched models reach 100% accuracy on ImageNet, which is impossible for ResNet-50. Additionally, it casts doubt on the results of Section 4.2 for the random noise stitched with Digit models. If I have misunderstood and the stitched network was evaluated on the noise dataset, I have two comments. First, the paper should clearly mention it in the main text. Second, these results do not provide much insight because the remaining receiver layers were trained for a classification with extracted features, which means stitching a trained feature extractor to them would be easy.\n2. A few experiments in the paper fail to distinguish between two possible explanations for the results: explanation via the properties of the similarity metric and explanation via the properties of a learning task. For example, it could indeed be possible that the representations of models trained on Stylized ImageNet are \"worse\" for usual ImageNet than those of models trained on usual ImageNet because the models trained on usual ImageNet capture additional texture information. If this is the case, model stitching indeed gave us valuable information about the learning task itself.\n3. The results in Section 4 lack a discussion of crucial reference points for comparison. Bansal et al. (2021) introduced two reference points: the performance of the receiver model and the performance of the randomly initialized model stitched with the receiver. In contrast, the current paper only analyzes the performance of the receiver model and does not discuss the performance of the randomly initialized model stitched with the receiver. The second reference point is crucial for the interpretation of the results. For example, if the stitch of a randomly initialized model achieves high accuracy, it means that the stitch is too powerful for the MNIST task, strengthening the concern expressed at the end of Section 4.\n4. The previous point also suggests that the paper should discuss the choice of stitching family for comparisons. For example, 1x1 convolutions might be indeed too powerful for meaningful comparison of representations for MNIST data. However, simple scalar rescaling of channels or strongly regularized 1x1 convolutions might still provide useful information about representations.\n5. Given that I have concerns about the experiments on MNIST data and stitches of random clustered noise with ImageNet receiver, I do not think the paper definitively shows that model stitching does not provide useful information about representation similarity.", "methods_and_evaluation_criteria": "I think the scope (ResNet-18 models on MNIST and ResNet-50 models on ImageNet) is sufficient. However, I think ResNet-18 model is too powerful for MNIST dataset. A better choice would be the LeNet-5 model or a simple convolutional model. Additionally, I think performing a hyper-parameter sweep to choose the stitching layer's learning rate would be more appropriate than using a fixed one.", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "I find some experiments not informative. I think the most informative designs were presented in Sections 4.2, 5.2, and 5.4. I would prefer the paper to either focus on these results or discuss other results' limitations more.", "relation_to_broader_scientific_literature": "The paper evaluates the limitations of a particular representation similarity metric. This investigation could be important for the deep learning theory and interpretability literature since representation similarity is an important concept in these fields.", "essential_references_not_discussed": "All essential references seem to be discussed.", "other_strengths_and_weaknesses": "I think the paper has some issues with writing and clarity.\n1. The discussion of noise at the end of Section 4.1 is confusing because the authors explain their construction of noise dataset only in Section 4.2.\n2. It is often hard to understand which datasets were used for model training, stitch training, and stitched model evaluation. For instance, I did not understand the textual definition of the sender's baseline and receiver's baseline in Section 5, and only managed to understand these notions after looking at Table D.1.\n3. I have trouble understanding the motivation and interpretation of some results. For example, I do not understand what we could infer from Section 6. This section does not directly study the considered model stitching metric, and basically only demonstrates qualitative results. At the same time, the finding that MNIST decoder will produce MNIST-like images does not seem surprising for me since this decoder can not produce anything else.", "other_comments_or_suggestions": "At the beginning of Section 5.4, the phrase \"to stitch a model trained to recognise bird songs to a pretrained ImageNet model\" contradicts the discussion after and Table 1. It should be the opposite \"to stitch a pretrained ImageNet model to a model trained to recognise bird songs\".", "questions_for_authors": "1. Do I correctly understand your methodology for model stitching?\n2. How did your stitch of random noise with ImageNet achieve 100% accuracy on the ImageNet dataset?\n3. Could you elaborate (compared to Section 7) on the cases where the model stitching could provide valuable information?\n4. Would you get different results if you used a less powerful stitch for the Colored MNIST dataset?\n5. Would you get different results if you used a less powerful architecture for the Colored MNIST dataset?\n6. How would you determine the quality of representations for a specific task without using the model stitching method?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies the approach of comparing model information content via model studying, using case studies to illustrate failure modes of this analysis. The majority of the reviewers agree that the paper offers a useful contribution, that helps to clearly illustrate the difficulty of interpreting stitching results. While one reviewer maintains some concerns about the methodology and the overall interpretation of the results — which I agree could be further clarified in the final version of the paper — on the whole the other reviewers and I agree that this is a valuable and clearly-presented contribution to the literature.
Decision: Accept (spotlight poster)