Harmony in Diversity: Merging Neural Networks with Canonical Correlation Analysis

Authors: stefan  horoi
albert manuel orozco camacho
eugene  belilovsky
guy  wolf
Venue: ICLR_cc_2024_Workshop_Re-Align
Published: 2024-02-08 18:08:06

Abstract: Ensembling multiple models enhances predictive performance by utilizing the varied learned features of the different models but incurs significant computational and storage costs. Model fusion, which combines parameters from multiple models into one, aims to mitigate these costs but faces practical challenges due to the complex, non-convex nature of neural network loss landscapes, where learned minima are often separated by high loss barriers. Recent works have explored using permutations to align network features, reducing the loss barrier in parameter space. However, permutations are restrictive since they assume a one-to-one mapping between the different models' neurons exists. We propose a new model merging algorithm, CCA Merge, which is based on Canonical Correlation Analysis and aims to maximize the correlations between linear combinations of the model features. We show that our method of aligning models leads to better performances than past methods when averaging models trained on the same, or differing data splits. We also extend this analysis into the harder many models setting where more than 2 models are merged, and we find that CCA Merge works significantly better in this setting than past methods.

Main paper: 
 introduction: 
 A key strategy for improving the performance and robustness of machine learning models involves utilizing multiple models which capture diverse, potentially complementary insights from data. Ensembles, which combine outputs from several trained models through averaging or majority vote at inference time, improve predictive performance but at the expense of increased memory and computational costs (Ho, 1995; Lobacheva et al., 2020) . An alternative, model fusion, merges parameters from multiple models into one by averaging them, reducing storage and computation costs but facing challenges in maintaining performance (Frankle et al., 2020; Stoica et al., 2024) . Indeed, while multiple good local minima can be found for a given model and task, these minima are most often separated by regions of high loss (Frankle et al., 2020) . Combining the parameters of trained models is likely to lead to a high loss region in parameter space, thus destroying the learned features.Linear mode connectivity which describes two optima connected by a linear low loss path in the parameter space (Frankle et al., 2020) , provides a straightforward way of combining models through linear interpolation of their parameters, the existing low loss path guaranteeing the preserving of performance. However, LMC is very rare in practice and is not guaranteed even for networks with the same initializations (Frankle et al., 2020) , and it is only made rarer by NN's permutation invariance. Entezari et al. (2022) conjectured that most SGD solutions can be permuted in such a way that they are linearly mode connected to most other SGD solutions. Many works in recent years have provided algorithms for finding permutations that successfully achieve LMC between pairs of SGD solutions, or at least significantly lower the loss barrier on the linear path between these solutions, further supporting this conjecture (Singh & Jaggi, 2020; Peña et al., 2023; Ainsworth et al., 2023) .However, there is nothing inherently stopping NNs from distributing computations that are done by one neuron in a model to be done by multiple neurons in the other model. Permutations would fail to capture this relationship since they can only account for one-to-one mappings between features. Furthermore, the focus of recent works has been mainly on merging pairs of models. However, if a similar function is learned by networks trained on the same task then it should be possible to extract the commonly learned features from not only two but also a larger population of models.Contributions Our main contributions are threefold: (1) We propose a new model merging method based on Canonical Correlation Analysis (Sec. 3) which we will refer to as "CCA Merge". This method is more flexible than past, permutation-based methods and therefore makes better use of the correlation information between neurons (Sec. A.1). (2) We compare our CCA Merge to past works and find that it yields better performing merged models across a variety of architectures and dataset. This is true in both settings where the models were trained on the same data (Sec. 4.1) or on disjoint splits of the data (Sec. 4.3). (3) We take on the difficult problem of aligning features from multiple models and then merging them. We find that CCA Merge is significantly better at finding the common learned features from populations of NNs and aligning them, leading to lesser accuracy drops as the number of models being merged increases (Sec. 4.2). 
related work: 
 "Easy" settings for model averaging Linear mode connectivity is hard to achieve in deep learning models (Frankle et al., 2020) and it seems that only models which are already very close in parameter space can be directly combined through linear interpolation. This is the case for snapshots of a model taken at different points during its training trajectory (Garipov et al., 2018; Izmailov et al., 2018) , multiple fine-tuned models with the same pre-trained initialization as is standard in NLP (Wortsman et al., 2022; Ilharco et al., 2023; Yadav et al., 2023) or models merged every couple of epochs during training as in federated learning settings (McMahan et al., 2017) . We emphasize that these settings are different from ours in which we aim to merge fully trained models with different parameter initializations and SGD noise (data order and augmentations). Merging multiple models Merging more than two models has only been explored thoroughly in the "easy" settings stated above (Wortsman et al., 2022; Jolicoeur-Martineau et al., 2023; Yadav et al., 2023) . The only works focused on providing feature alignment methods which have also applied these methods to the multiple model merging setting are Ainsworth et al. (2023) with their "Merge-Many" algorithm and Singh & Jaggi (2020). However their results for merging multiple models are mainly relegated to the appendix and focus on easy settings such as MLPs on MNIST or they fine-tune the resulting merged model. We extend this line of work to more challenging settings, using more complex model architectures, we report the merged models accuracies directly without fine-tuning and make this a key focus in our work. Model merging beyond permutations We note that the two model merging methods based on optimal transport Singh & Jaggi (2020) and Peña et al. (2023) can also align models beyond permutations. However, in Singh & Jaggi (2020) this only happens when the two models being merged have different numbers of neurons at each layer, which is not the case for the majority of their experiments. The method proposed by Peña et al. (2023) isn't constrained to finding binary permutation matrices but binarity is still encouraged through the addition of an entropy regularizer. For a more complete account of related works we direct the reader to the Appendix. 
using cca to merge models: 
 Merging Models Consider two deep learning models A and B with the same architecture. Let {L M i } N i=1 be the set of layers of model M ∈ {A, B} and let X M i ∈ R m×ni denote the set of outputs of the i-th layer of model M in response to m given inputs. We assume X M i is centered. We are interested in merging the parameters from models A and B in a layer-by-layer fashion. In practice, it is often easier to keep model A fixed and to find a way to transform model B such that the average of their weights can yield good performance. Mathematically, we are looking for invertible linear transformations T i ∈ R ni×ni which can be applied at the output level of model B layer i parameters to maximize the "fit" with model A's parameters and minimize the interpolation error. The inverse transform is then applied at the input level of the next layer to keep the flow of information consistent inside a given model. With these transformations, σ being the non-linearity and W, b being the weight and bias parameters, the output of the transformed layer i of B becomes:x B i = σ(T i W B i T -1 i-1 x B i-1 + T i b B i )After finding transformations {T i } i=1 for every merging layer in the network we can merge the two model's parameters:EQUATIONCanonical Correlation Analysis Canonical Correlation Analysis (CCA) is a statistical method aiming to find relations between two sets of random variables. Suppose we have two datasets X and Y both of sizes n × d, where n is the number of instances or samples, and d is the dimensionality or the number of features. Further, suppose that these datasets are centered so that each column has a mean of 0. CCA finds vectors p X and p Y in R d such that the projections Xp X and Y p Y have maximal correlation and norm 1. By iteratively finding such vectors, with the added constraint that each new vector must be orthogonal to the other ones, CCA can find the full projection matrices P X and P Y aligning XP X and Y P Y . For more details we direct the reader to De Bie et al. (2005) .CCA Merge: Merging models with CCA To merge models using CCA we simply apply the CCA algorithm to the two sets of activations X A i and X B i to get the corresponding projection matrices P A i and P B i . Using the framework described above of matching model B to model A, which is consistent with past works, we can defineT i = P B i P A i -1 ⊤. The transpose here is to account for the fact that T i multiplies W i on the left while the P i s were described as multiplying X i on the right. This transformation can be thought of as first bringing the activations of model B into the common, maximally correlated space between the two models by multiplying by P B i and then applying the inverse of P A i to go from the common space to the embedding space of A. The averaging of the parameters of model A and transformed B can then be conducted following Eq. 1. 
models merged with cca merge achieve better performance: 
 In Table 1 we show the test accuracies of merged VGG11 (Simonyan & Zisserman, 2015) , ResNet20 and ResNet18 (He et al., 2016) models of different widths trained on CIFAR10 (Krizhevsky & Hinton, 2009) , CIFAR100 and ImageNet200 (Russakovsky et al., 2015) (full-sized images but training and evaluating only on 200 of the 1k classes) for CCA Merge and multiple other popular model merging methods. The models were trained either using the one-hot encodings of the labels or the CLIP (Radford et al., 2021) embeddings of the class names as training objectives. We report the average accuracies of the base models being merged under the label "Base net avg." (i.e. each model is evaluated individually and their accuracies are then averaged) as well as the accuracies of ensembling the models (the logits of the different models are averaged and the final prediction is the argmax). Ensembling is considered to be the upper limit of what model fusion methods can achieve. Also, since the models being merged were trained on the same data, we do not expect the merged models to outperforms the base ones in this particular setting.We compare CCA Merge with the following methods: Direct averaging: averaging the models' weights without any neuron alignment; Permute: permuting model weights to align them, the permutation matrix is found by solving the linear sum assignment problem consisting of maximizing the sum of correlations between matched neurons (Li et al., 2015; Tatro et al., 2020) ; Matching Weights: main method from Ainsworth et al. (2023) ; ZipIt!: model merging method proposed by Stoica et al. (2024) . For ResNets, we recompute the BatchNorm statistics after the weight averaging and before evaluation as suggested by Jordan et al. (2023) to avoid variance collapse.VGG11 models merged with CCA Merge have significantly higher accuracies than models merged with any other method, and this is true across all model widths considered. Differences in accuracy ranging from 10% (×8 width) up to 25% (×1 width) can be observed between CCA Merge and the second-best performing method. Furthermore, CCA Merge is more robust when merging smaller width models, incurring smaller accuracy drops than other methods when the width is decreased from ×8 to ×1; 1.71% drop for CCA Merge versus 18.34% for Matching Weights and 8.19% for Permute. Lastly, CCA Merge seems to be more stable across different initializations, the accuracies having smaller standard deviations than all other methods for the same width except for Matching Weights for ×8 width models. We note that for VGG models with width multipliers above ×2 and ResNet18 on ImageNet200, we ran into out-of-memory issues when running ZipIt!, which is why those results are not present. The same conclusions seem to hold for ResNet20 trained on CIFAR100 and ResNet18 on ImageNet200, although the differences in performance here are less pronounced. For ImageNet, the top 5 accuracy of models merged with CCA Merge is remarkably close to the accuracy of model ensembles, the peak of attainable performance.ResNet20×8 For both VGG and ResNet architectures and for all considered datasets the added flexibility of CCA Merge over permutation-based methods seems to benefit the merged models. Aligning models using linear combinations allows CCA Merge to better model relationships between neurons and to take into account features that are distributed across multiple neurons.In this section, we present our results related to the merging of many models, a significantly harder task. This constitutes the natural progression to the problem of merging pairs of models and is a more realistic setting for distributed or federated learning applications where there are often more than 2 models. Furthermore, aligning populations of neural networks brings us one step closer to finding the common learned features that allow different neural networks to perform equally as well on complex tasks despite having different initializations, data orders, and data augmentations.The authors of Ainsworth et al. (2023) introduced "Merge Many", an adaptation of Matching Weights for merging a set of models. A simpler way of extending any model merging method to the many models setting is to choose one of the models in the group as the reference model and to align every other network in the group to it. Then the reference model and all the other aligned models can be merged. It is by using this "all-to-one" merging that we extend CCA Merge, Permute, and Matching Weights to the many model settings. ZipIt! is naturally able to merge multiple models since it aggregates all neurons and merges them until the desired size is obtained.In Fig. 1 we show the accuracies of the merged models as the number of models being merged increases. For both VGG and ResNet architectures aligning model weights with CCA continues to yield better performing merged models. In fact, models merged with CCA Merge applied in an allto-one fashion maintain their accuracy relatively well while the other ones see their accuracies drop significantly. In the VGG case, the drop for other methods is drastic, all merged models having less than 20% accuracy when more than 3 models are being merged while CCA Merge suffers a drop in accuracy of less than 3% when going from 2 to 5 models, staying around the 80% mark. We also note that Merge Many performs only slightly better than its 2 model counterpart (Matching Weights applied in an all-to-one fashion). For ResNets, the accuracy of models merged with Permute drops by 15% when going from merging 2 models to 20. CCA Merge on the other hand is a lot more robust, incurring a less than 4% drop in accuracy even as the number of models is increased to 20.These results suggest that CCA Merge is significantly better than past methods at finding the "common features" learned by groups of neural networks and aligning them. The limitations of permutation-based methods in taking into account complex relationships between neurons from different models are highlighted in this context. Here it is harder to align features given that there are more of them to consider and therefore easier to destroy the features when averaging them.In this section we consider the more realistic setting where the models are trained on disjoint splits of the data, therefore they're expected to learn (at least some) different features. Such a set-up is (1) 80%-20% 2 : ResNet20×8 trained on 3 different splits of CIFAR100 -Accuracies and standard deviations from 4 different merges of 2 models are presented. When the models being merged have learned different features from disjoint sets of the training data but with all the classes (splits 1 and 2) CCA Merge is the only model merging method which outperforms the average of the base models. In the case where the models being merged were trained on disjoint subsets of the classes (split 3) CCA Merge still outperforms past model merging methods except for ZipIt!. natural in the context of federated or distributed learning. We consider ResNet20 models trained on 3 different data splits of the CIFAR100 training dataset. The first (1) data split is the one considered in Ainsworth et al. (2023) and Jordan et al. (2023) where one model is trained on 80% of the data from the first 50 classes of the CIFAR100 dataset and 20% of the data from the last 50 classes, the second model being trained on the remaining examples. In the second (2) data split we use samples from a Dirichlet distribution with parameter vector α = (0.5, 0.5) to subsamble each class in the dataset and create 2 disjoint data splits, one for each model to be trained on. Lastly, with the third (3) data split we consider the more extreme scenario from Stoica et al. (2024) where one model is trained on 100% of the data from 50 classes, picked at random, and the second one is trained on the remaining classes, with no overlap. For this last setting, in order for both models to have a common output space they were trained using the CLIP Radford et al. (2021) embeddings of the class names as training objectives. In Table 2 we report mean and standard deviation of accuracies across 4 different model pairs.For the first two data splits CCA Merge outperforms the other methods, beating the second best method by 4% and 2% on the first and second data splits respectively. For the third data split CCA Merge is the second best performing method after ZipIt!. However, we note that the comparison with ZipIt! is somewhat unfair, ZipIt! was designed for this specific setting and it allows the merging of features from the same network to reduce redundancies, thus making it more flexible than the other methods which only perform "alignment". In all cases CCA Merge outperforms or is comparable with the base models average indicating that, to some extent, our method successfully combines different learned features from the two models. 
discussion and conclusion: 
 Recent model fusion successes exploit inter-model relationships between neurons by modeling them as permutations before combining them. Here, we argue that, while assuming a one-to-one correspondence between neurons yields interesting merging methods, it is rather limited as not all neurons from one network have an exact match with a neuron from another network. Our proposed CCA Merge takes the approach of linearly transforming model parameters beyond permutation-based optimization. This added flexibility allows our method to outperform recent competitive baselines when merging pairs of models trained on the same data or on disjoint splits of the data (Tables 1 and 2 ). Furthermore, when considering the harder task of merging many models, CCA Merge models showcase remarkable accuracy stability as the number of models merged increases, while past methods suffer debilitating accuracy drops. This suggests a path towards achieving strong linear connectivity between a set of models, which is hard to do with permutations (Sharma et al., 2024) . value of one), suggesting that the feature learned by each neuron z A i from model A is distributed across multiple neurons from B -namely, those having high correlations -as opposed to having a single highly correlated match.Given the flexibility of CCA Merge, we expect it to better capture these relationships between the neurons of the two networks. We recall that CCA Merge computes a linear transformation T that matches to each neuron z A i a linear combinationz A i ≈ n j=1 T ij z B j of the neurons in B.We expect the distribution of the coefficients (i.e., elements of T ) to match the distribution of the correlations (C ij elements), indicating the linear transformation found by CCA Merge adequately models the correlations and relationships between the neurons of the two models. For each neuron z A i , we select its top k-th, for k ∈ {1, 2}, correlation from the i-th row of C and its top k-th coefficient from the i-th row of T and we plot a histogram of these values for all neurons {z A k } n k=1 in Fig. 3 . Indeed, the distributions of the correlations and those of the CCA Merge coefficients are visually similar, albeit not fully coinciding. To quantify this similarity we compute the Wasserstein distance between these distributions, normalized by the equivalent quantity if the transformation were a permutation matrix. For a permutation matrix, the top 1 values would be of 1 for every neuron z A i and all other values would be 0. We can see that CCA Merge finds coefficients that closely match the distribution of the correlations, more so than simple permutations, since the ratio of the two distances are 0.15 and 0.04, respectively, for top 1 values in the two considered layers and 0.35 and 0.23 for top 2 values. 2017) proved theoretically that one-layered ReLU neural networks have asymptotically connected level sets, i.e. as the number of neurons increases two minima of such a network are connected by a low loss path in parameter space. Garipov et al. (2018) and Draxler et al. (2018) explore these ideas empirically and introduce the concept of mode connectivity to describe ANN minima that are connected by nonlinear paths in parameter space along which the loss remains low, the maximum of the loss along this path was termed the energy barrier, and both works proposed algorithms for finding such paths. Garipov et al. (2018) further proposed Fast Geometric Ensembling (FGE) as a way to take advantage of mode connectivity by ensembling multiple model checkpoints from a single training trajectory. Frankle et al. (2020) introduced the concept of linear mode connectivity which describes the scenario in which two ANN minima are connected by a linear low loss path in parameter space. They used linear mode connectivity to study network stability to SGD noise (i.e. different data orders and augmentations). They found that at initialization, networks are typically not stable, i.e. training with different SGD noise from a random initialization typically leads to optima that are not linearly mode connected, however early in training models become stable to such noise.Model merging More recently, Entezari et al. (2022) have conjectured that "Most SGD solutions belong to a set S whose elements can be permuted in such a way that there is no barrier on the linear interpolation between any two permuted elements in S" or in other words most SGD solutions are linearly mode connected provided the right permutation is applied to align the two solutions. They then perform experiments that support this conjecture and also empirically establish that increasing network width, decreasing depth, using more expressive architectures, or training on a simpler task all help linear mode connectivity by decreasing the loss barrier between two optima. Many other works seem to support this conjecture. For example, Singh & Jaggi (2020) and Peña et al. (2023) propose optimal transport-based methods for finding the best transformation to match two models. Ainsworth et al. (2023) propose an algorithm for finding the optimal permutation for merging models based on the distances between the weights of the models themselves. Jordan et al. (2023) exposes the phenomenon in which interpolated deep networks suffer a variance collapse in their activations leading to poor performance. They propose REPAIR which mitigates this problem by rescaling the preactivations of interpolated networks through the recomputation of BatchNorm statistics."Easy" settings for model averaging Linear mode connectivity is hard to achieve in deep learning models. Frankle et al. (2020) established that even models being trained on the same dataset with the same learning procedure and even the same initialization might not be linearly mode connected if they have different data orders/augmentations. It seems that only models that are already very close in parameter space can be directly combined through linear interpolation. This is the case for snapshots of a model taken at different points during its training trajectory (Garipov et al., 2018; Izmailov et al., 2018) or multiple fine-tuned models with the same pre-trained initialization (Wortsman et al., 2022; Ilharco et al., 2023; Yadav et al., 2023) . This latter setting is the one typically considered in NLP research. Another setting that is worth mentioning here is the "federated learning" inspired one where models are merged every couple of epochs during training (Jolicoeur-Martineau et al., 2023) . The common starting point in parameter space and the small number of training iterations before merging make LMC easier to attain.We emphasize that these settings are different from ours in which we aim to merge fully trained models with different parameter initializations and SGD noise (data order and augmentations).Merging multiple models Merging more than two models has only been explored thoroughly in the "easy" settings stated above. For example Wortsman et al. (2022) averages models fine-tuned with different hyperparameter configurations and finds that this improves accuracy and robustness. Jolicoeur-Martineau et al. (2023) averages the weights of a population of neural networks multiple times during training, leading to performance gains. On the other hand, works that have focused on providing feature alignment methods to be able to merge models in settings in which LMC is not trivial have mainly done so for 2 models at the time (Singh & Jaggi, 2020; Ainsworth et al., 2023; Peña et al., 2023; Jordan et al., 2023) . An exception to this is Git Re-Basin (Ainsworth et al., 2023) which proposes a "Merge Many" algorithm for merging a set of multiple models by successively aligning each model to the average of all the other models. However, results obtained with this method, which they use to merge up to 32 models, are relegated to the appendix and only concern the very simple set-up of MLPs on MNIST. Singh & Jaggi (2020) also consider merging multiple models but either in a similarly simple set-up, i.e. 4 MLPs trained on MNIST, or they fine-tune the resulting model after merging up to 8 VGG11 models trained on CIFAR100. We extend this line of work to more challenging settings, using more complex model architectures, we report the merged models accuracies directly without fine-tuning and make this a key focus in our work.Model merging beyond permutations We note that the two model merging methods based on optimal transport Singh & Jaggi (2020); Peña et al. ( 2023) can also align models beyond permutations. However, in Singh & Jaggi (2020) this only happens when the two models being merged have different numbers of neurons at each layer. When the models have the same number of neurons the alignment matrix found by their method is a permutation, as such the majority of their results are with permutations. The method proposed by Peña et al. ( 2023) isn't constrained to finding binary permutation matrices but binarity is still encouraged through the addition of an entropy regularizer. Furthermore, our CCA based method is different in nature from both of these since it is not inspired by optimal transport theory.CCA in deep learning Canonical Correlation Analysis is a very popular statistical method used in many fields of science De Bie et al. (2005) . In the context of deep learning, CCA has been used to align and compare learned representations in deep learning models Raghu et al. (2017) ; Morcos et al. (2018) , a task which is very similar to the feature matching considered by model merging algorithms. 
 
 Appendix: Our code is publicly available at https://github.com/shoroi/align-n-merge 



Reviews: review: title: title: Clear, well-motivated, good experiments. Held back by missing code.
main_review: main_review: Summary: The paper proposes a new model merging method based on CCA, “CCA Merge”. Contrary to prior methods, the proposed method is more flexible by allowing alignment via linear transformations instead of permutations. CCA Merge outperforms prior methods both when merging two and especially multiple models.

Strengths: 
- The objective of the paper is clear and well-motivated.
- The claims regarding performance are well supported. However, I was surprised that the gap to pure permutation is quite low in some cases.
- Comprehensive related work section that makes the paper accessible to a wider audience.

Weaknesses:
- No code is released as far as I see. Releasing the code would make the work more useful to the community and guarantee reproducibility.
- The introduction portraits model merging in context of improving performance similar to ensembles. This is in contrast to the results, where merged models are worse than single models even (Table 1). Do I understand correctly that improving performance would be the end goal, but we are not there yet? I think clarification could help here.

Minor comments and questions: 
- The merging transformation $T_i$ stems from the optimal matching between the _activations_, but $T_i$ is then applied on the pre-activations. Would it not make more sense to apply CCA on the outputs of a layer before activation function?
- Please describe the training with CLIP embeddings mentioned in Section 4.3.
- Would you say your approach is connected to model stitching [1]? Training a linear stitching layer between layers at equal depth could maybe give you a transformation similar to the one you get via CCA. 

[1] Lenc, Karel, and Andrea Vedaldi. "Understanding image representations by measuring their equivariance and equivalence." Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.
strength_weakness: strength_weakness: - The proposed method is novel and has solid evidence.
- Overall high quality paper.
title: title: A good paper, recommended for acceptance
main_review: main_review: This paper propose a novel method, called CCA Merge, for merging different models. The method is based on the idea of Canonical Correlation Analysis, a statistical method traditionally employed to discern the maximal correlation between linear composites derived from two multivariate datasets. The application of CCA in this context is both novel and reasonable.

The paper is easy to follow and provides good presentations. To my understanding, merging different models on the same task can be a promising way to get better performance. Merging in such a "direct" way can be effective, but meanwhile challenging due to the complexity entailed in feature spaces. 

The proposed method is simple, elegant and seems interesting. From the results, it outperforms the current merging methods to a certain extent. Nevertheless, we should also note a saline limitations of these methods, which concerns their inconsistent ability to guarantee performance augmentation after merging. Merged models may be worse compared to the separate unmerged models, which makes the proposed method less useful. Additionally, we encourage the authors can give theoretical backups in their future work.
strength_weakness: strength_weakness: A good paper can provide interesting insights in regards to the merging model. Meanwhile, the proposed method is simple and interesting, which can outperform the contemporary methods.
title: title: Novel alignment method with limited theoretical explanation
main_review: main_review: This paper proposes a new model merging method based on Canonical Correlation Analysis (CCA Merge). Compared to existing alignment methods, CCA Merge yields better performing merged models for VGG and ResNet architectures trained on CIFAR10, CIFAR100, and ImageNet200. CCA Merge also performs significantly better at aligning multiple models. 

**Strengths**
- The writing is very clear. The limitations of ensembling, model fusion, and permutation-based alignment methods are explained well. The use of CCA for alignment is well motivated and novel.
- Appendix A.1 provides good intuitions of why permutation is not sufficient. 
- Experiment results are impressive. The proposed method produces merged models with significantly better accuracy than all baselines, especially for the VGG11 model.
- The paper addresses the problem of improving alignment among trained models, which is highly relevant to the community at the workshop. In particular, the experiments on merging multiple models demonstrates that CCA is better than past methods at finding the common features learned by groups of neural networks. 

**Weaknesses**
- The methodology section could benefit from more explanations and details about CCA. 
- While some intuitions are provided in the appendix, the paper does not give a theoretical justification the advantage of CCA over permutation or guarantees of the accuracy of the merged model.
- The end of the first paragraph in section 4.1 states that the merged models are not expected to outperform the base ones. In this case, why do we need the merged model? Can we just use the best base model instead?
strength_weakness: strength_weakness: The proposed method is novel and performs significantly better at alignment tasks than all baselines.

Comments: comment: We thank the reviewers for their thoughtful comments and insightful questions. Their constructive criticism will be taken into consideration to help us improve future iterations of our work.