$C^2M^3$: Cycle-Consistent Multi-Model Merging
Authors: Donato Crisostomi
Marco Fumero
Daniele Baieri
Florian Bernard
Emanuele Rodolà
Venue: NeurIPS 2024 poster (2024)
Keywords: model merging
linear mode connectivity
deep learning
Primary area: deep_learning_architectures
TL;DR: Based on the conjecture that all modes found by SGD live in the same basin up to permutations, we propose a novel weight-matching procedure based on the Frank-Wolfe algorithm that ensures cycle consistency of the permutations.

Abstract: In this paper, we present a novel data-free method for merging neural networks in weight space. Our method optimizes for the permutations of network neurons while ensuring global coherence across all layers, and it outperforms recent layer-local approaches in a set of challenging scenarios. We then generalize the formulation to the $N$-models scenario to enforce cycle consistency of the permutations with guarantees, allowing circular compositions of permutations to be computed without accumulating error along the path. 
    We qualitatively and quantitatively motivate the need for such a constraint, showing its benefits when merging homogeneous sets of models in scenarios spanning varying architectures and datasets. We finally show that, when coupled with activation renormalization, the approach yields the best results in the task.

C²M3: Cycle-Consisten Multi-Model Merging
Donato Crisostomi Sapienza University of Rome crisostomion oddidi..midd..................ii uniromal...
Marco Fumero
Institute of Science and Technology Austria fumero@di. uniromal....
Daniele Baieri Sapienza University of Rome baieriedi. uniromal. it
Florian Bernard University of Bonn fb@uni-be mommi-bonn..o de
Emanuele Rodolà
Sapienza University of Rome rodola@di. uniromal.. it
Abstract
this paper. we present a novel data-free method for merging neural networks in weight space. Differently from most existing works, our method optimizes for the permutations of network neurons globally across all layers. This allows us to enforce cycle consistency of the permutations when merging 2 3 models, allow- ing circular compositions permutations to be computed without accumulating error along the path. We qualitatively and quantitatively motivate the need for such a constraint, showing its benefits when merging sets of models scenarios spanning varying architectures and datasets. We finally show that, when coupled with activation renormalization, our approach yields the best results in the task.
Introduction
In the early days of deep learning, modes parameters corresponding to local minima of the loss landscape were considered to be isolated. Being depicted as points the bottom of convex valleys, they were thought to be separated by high-energy barriers that made the transition between them impossible. However, a series of recent works have gradually challenged this perspective, first showing that modes can be actually connected by paths of low energy [10, 14], and later that, in some cases, these paths may even be linear [13]. While linear paths in [13] could only be obtained after training the equally-initialized models for a few epochs, follow-up work [1 speculated that the isolation of modes is result of the permutation symmetries of the neurons. In fact, given layer We of a fixed network A, large number of permuting the neurons of W, by some permutation and then anti- antiipermuting the columns of the subsequent layer W This intuition led to the conjecture that all modes lie in the same convex region of the parameter space, denoted as basin, when taking into account all possible permutations of the neurons of network. This motivated series of works trying to align different modes by optimizing for the neuron permutations [1. 21, 29, 36]. This has strong implications for model merging, where different models, possibly trained with different initializations [I, 29, 34] or different datasets and tasks [1, 36], are aggregated into single one. this work, we focus on the data-free setting, aligning networks based on some similarity function that is computed directly over the neurons themselves. To this end, we follow Ainsworth et al. [1] and formalize the problem of model merging previous approaches while allowing global constraints to be enforced.
networks can be obtained by
assignment problem, proposing new algorithm that is competitive with
The problem We investigate the problem of merging > 2 models, noting that existing pairwise approaches such as [1] do not guarantee cycle consistency of the permutations (see Figure 1). As shown in Figure 2b and Figure 2a, going from model A to model C' through model B, and then mapping back to A, results in a different model than the starting one - specifically, the target
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

A
A
P4
(PA)T
PAC
PBA
p8
(PO)T
PC
(PB)
B
C
B
PCB
Figure I: Cycle-Consistent Multi-Model Merging over three models A, B, C. Left: existing methods seek pairwise permutations that map between models; note that p4c pCB Pn + in general, unless this explicitly enforced. Right: our method computes permutations PA, pn, pC from each model to universe U, such that pairwise permutation PBd mapping A to B can be obtained as PBA pB (PA) This way, cycle-consistency is enforced by design and PAC o pCB PBA T.
model ends up in completely different basin. More formally, for these methods, the composition of permutations along any cycle does not result in the identity map. This also holds for the n = 2 case, where the permutations optimized to align model A to model B are not guaranteed to be the inverse of those mapping B to A: this makes the alignment pipeline brittle, as depends on an arbitrary choice of mapping direction.
Contribution To address this issuc, we introduce a novel alignment algorithm that works for the general case with > 2 models, while guaranteeing cycle consistency. The key idea is to factorize each permutation mapping B to A as pAB = PA (PD) where (P")* maps B to a common space denoted as universe, and P^ maps from the universe back to A. This formulation ensures cycle consistency by design, as any cyclic composition of such permutations equals the identity.
Our numerical implementation is based on the Frank Wolfe algorithm [12], and optimizes for the permutations of all the layers simultaneously at each step, naturally taking into account the inter-layer dependencies in the process. This desirable property is in contrast with other approaches such as Ainsworth et al. [1], which seek the optimal permutations for each layer separately, and thus can not ensure coherence across the entire network.
We run an extensive comparison of our approach with existing ones both in the standard pairwise setting and in merging n > 2 models, spanning a broad set of architectures and datasets. We then quantitatively measure the influence of architectural width, confirming the existing empirical evidence on its role in linear mode connectivity. Further, we assess how the performance of the merged model depends on the number of models to aggregate, and show that the decay graceful. We finally analyze the basins defined by the models when mapped onto the universe, and investigate when and to what extent these are linearly connected.
Wrapping up, our contributions are four-fold:
We propose new data-free weight matching algorithm based on the Frank-Wolfe algo- rithm [12] that globally optimizes for the permutations of all the layers simultaneously;
We generalize to the case of n 2 2 models, enforcing guaranteed cycle-consistency the permutations by employing a universal space as a bridge;
We leverage the multi-model matching procedure for model merging, using the universal space as aggregation point;
We conduct an extensive analysis showing how the merge is affected by the number of models, their width and architecture, as well as quantitatively measuring the linear mode connectivity in the universe basin.
Finally, to foster reproducible research in the field, we release a modular and reusable codebase containing implementations of our approach and the considered baselines.!

---

Acctiaay
Lass
Permulation
Gii Basin
C
0.6
d (A, PA-B--C-A(A))
41.07
0.0
(B, PB-C-AA-B(B)
41.18
0.0
d (CAA-B--))))
41.19
0.0
0.0
1.0
Re-Basin
Ca
(a) Loss and accuracy curves for model A and the model mapped back after cyclic permutation. Mad- els eyclically permuted with Git Re-Basin end up a different basin than the one they started from.
(b) Accumulated obtained when cyclically permuting models B and C in Figure I. PA-B-C-AA refers to the composition PAc Po РиΛ and d(-) is the & loss.
Figure 2: Existing methods accumulate error when cyclically mapping a model through a series of permutations, while C22 3 correctly maps the model back to the starting point.
2 Background
Mode connectivity introduced Section 1, mode connectivity studies the geometry of the loss landscape with a particular interest on the regions corresponding to local minima. Following Frankle al. [13], we assess the connectivity for two given modes by computing their loss barrier:
Definition 2.1. (Loss barrier) Given two points O,,, Og and a loss function C such that L (9A) x C (())). the loss barrier is defined as
1 (C(O))+ -
max L ((1 - A)OA + 10 AE
Intuitively, this quantity measures the extent of the loss increase when linearly moving from the basin of mode the other. When two modes share the same basin, the loss does not increase at all and results in a barrier close to zero.
Weight-space symmetries Following the rich line of works on mode connectivity and model merging [1, 11, 13, 29, 36], we start from the essential insight of neuron permutation invariance in neural networks. Let us focus the simple case of Multi-Layer Perceptrons (MLP), where can write the computation for an intermediate layer We E Rd x as ž4+1 (Wezz + be), with zg being the input at the f-layer and a denoting an element-wise activation function. For the sake of clear exposure, we consider the bias by = 0 in the following. If apply a permutation matrix P E P to the rows of the We matrix (i.e. the neurons), we obtain 2/41 = a (PW, Being an element-wise operator, commutes with P and can be neglected wlog. Since 2/441 * zς when P I, we can still nullify the effect of the permutation by anti anti-permuting the columns of the subsequent layer for the inverse permutation of P, i.e. PT. In fact,
2/42 We+1 4/ = W4+1 P e 2E+2
making pairs of models that only differ by permutation of the neurons de facto functionally equivalent. Given the enormous number of such permutations, it stands to reason that the resulting weight-space symmetries act major factor in the isolation of modes.
Solving for the permutation Given the above considerations, Entezari et al. [11] speculated that all models end up single basin after having accounted for permutation symmetries. Assuming this to hold at least in practical cases, Ainsworth et al. [1] proposed simple algorithm to find the permutations matching two models by maximizing local version of the sum of bi-linear problems:
arg max (PrEP)
Σ (A P.W.PT.) 8=1
(I)

---

with Po := I. Noting that Equation (1) is NP-hard, Ainsworth et al. [1] tackle this problem by considering one layer at a time, relaxing the bi-linear problems to a set linear ones that can be efficiently solved with any Linear Assignment Problem (LAP) solver, e.g., the Hungarian algorithm. This layer-wise linearization of the objective function, however, corresponds to high variance in the results that depend on the random order of the layers during optimization. See Table 7 for an empirical evaluation confirming this issue.
Renormalizing the activations Notwithstanding the quality of the obtained matching, the loss barrier can still be high due to the mismatch in the statistics of the activations. fact, REPAIR [21] empirically shows the presence of decay in the variance of the activations after the interpolation. They further show that the loss can be drastically reduced by "repairing" the mean and variance of the activations, forcing the statistics of the merged network to interpolate those of the endpoint networks. We refer the reader to Appendix A.4 for an in-depth explanation.
3 Approach
We now propose a novel algorithm to tackle the weight matching problem, first introducing its formulation in the pairwise case and then generalizing it to match and merge larger number n of models in a cycle-consistent fashion.
Pairwise matching As we have seen, the NP-hardness of Equation (1) demands for relaxation of the problem to be tackled. Differently from Ainsworth al. [1], we opt to maintain the objective global with respect to the layers and instead iteratively optimize its linear approximation via the the
Frank- Wolfe algorithm [12]. This procedure requires the computation of the gradient Equation (1) with respect each permutation P., thus we have to account for two contributions for each P., i.e., its gradient from permuting the rows of We and the one from permuting the columns of Wi+1:
Vp P.-.--(W T + (WA1) W
(2)
from permuting rows
from permurting columns
The Frank-Wolff algorithm then uses the gradient iteratively update the solution by linearly interpolating between the current solution and the projected gradient. We refer to Lacoste-Julien [24] for theoretical guarantees of convergence. The full algorithm is reported in Appendix A.2.
Generalization to n models In order to generalize to n models, we jointly consider all pairwise problems
a ΣΣΣ
WP. (PPP
arg max EP
where the superscript pq indicates that the permutations maps model q model p. with P/" = I. In order to ensure cycle consistency by construction we replace the quadratic polynomial by a fourth- order polynomial. Dropping the layer subscript for the sake of clear exposure, replace the pairwise matchings PM in the objective of Equation (3) by factorizing the permutations into object-to- to-universe
matchings PM pP (P9)² so that each model can be mapped back and forth to common universe u with permutation and its transpose, allowing to map model q to model p by composition of (P")* (q + u) and PP (u + p). This way, the objective of Equation (3) becomes
ΣΣ WWT. (P((P, potq 1=1
Σ Σ (TP)) WWPP (P?P (4)
po 1=1
As stated by Theorem 3.1, the permutations obtain using Equation (4) are cycle consistent. We refer the reader to Bernard et al. [5] for the proof and a complete discussion of the subject. Theorem 3.1. Given set of models Pu..... Pn and object-to-uee -u-uiveree permutations P." computed via Equation (4), the pairwise correspondences defined by prid, p. o (P.²)² are cycle-consistent, i.e.
.r.
P.iPs ... pPoPa pPaPL = I
for all layer indices i, 2 S

---

Similarly to the pairwise case, the approach requires computing the gradients for the linearization. This time, however, each r has four different contributions: one from permuting the rows of its corresponding layer, one from anti-permuting the columns of the subsequent layer, and two other contributions that arise from the symmetric case where A becomes B. In detail,
PA
Vious Pe
Vools
VlOWs, PA
+ vools
P/A
(5)
where
TOWS VIONS
W; WAP.A,,PP. (W: P
Vook
(W P (P W P.
tooss PA
pPP (W PA
Dook, P/
(W T Tpt (PA1) T W4 Pt
See Algorithm I for a complete description of the procedure.
Algorithm 1 Frank-Wolfe for n-Model Matching
Require: Weights of n. models M
tolerance >> 0
Ensure: Approximate solution to Equation (4)
1: p iddentity matrices
2: repeat
3: for (p.g) E [l,,.................. [1] [1, m do
4: for i = 1 to L do
5:
pp.k p + permutations over rows and columns of W?" respectively
pg.k: f-1 + permutation over rows and columns of W respectively
6:
7:
V pp.k f (WE+)) (P2+2) W Pi
8:
V p f 11 PP41 (P W E+1 P
9:
end for
10:
end for
11:
for Pk E pk do
11; LAP(V f) end for
14:
a - line search f. P. III
15: for Ph E pk do
16: pk- = (1--((1aa) P/ a 11;
end for
18:
until [[, B, B.Pk-l)
B, B() <e
19:
return pk
Merging in the universe space Looking at the loss landscape resulting from interpolating models in Figure 3, we see that the loss curves are much lower when the models are interpolated in the universe space. fact, the originally disconnected modes end up the universe, making it suitable to average the models. Therefore, our merging method aggregates the models by taking the mean of the weights in the universe space, as detailed in Algorithm
same basin when mapped onto
Algorithm 2 C²M Cycle-Consistent Multi Model Merging
Require: N models AI...., AN with layers
Ensure: merged model M
1: {P...... Px} + Frank-War Traak--ooff(MM,..... Wolfc(MM.........
2: for i 1 to N do
3: M.un +- map to uni
4: end for
5: Muni + 1 EM M
N
6: return Muni
5

---

C
HC
6
m
(a) ReaNet20 over CIFAR100.
(b) MLP over MNIST.
Figure 3: 2D projection of the loss landscape when matching three modes A, θ θc; the models ((OA), A), i (OB), (Oc)) are their resulting images in the universe, and lie in the same basin. Red zones indicate low-loss regions (typically basins), while blue zones indicate high-loss ones.
4
Experiments
We now evaluate the quality of proposed framework both in matching models and in the subse- quent merging operation. Approaches suffixed with a indicate the application of HEPAIR.
Matching and merging two models As described Section 3, our formalization n = models. In this case, the energy given by Equa- tion (1) and the permutations are not factorized. We com- pare the performance of our approach against the Git Re he-Basin algorithm [1] and the naive baseline that ag- gregates the models by taking an unweighted mean on the original model weights without applying any permu- tation. this setting, our method performs on par with the oftte-of-hhe-rt. Differently from the latter, however, we do not depend on the random choice of layers, as the optimization is performed over all layers simultancously, As presented in Figure 4, this results in Git Re-Basin exhibiting variations of up 10% accuracy depending on the optimization seed, while our method shows zero variance. We refer the reader to Appendix B. for thor- ough evaluation of C² C2M" 33 over set of different datasets and architectures. In summary, our approach able to match two models with the same accuracy as the state-of-fe oftte-of-the-rr, while being deterministic and independent of the random choice of layers.
readily be used to match
0.50
0.77
Accunne
0.70
0.77
0.72
Saed
Duxta
teoln
Fronk Wodee
troin
Ho-Hoii
Faank Welle
Figure 4: Accuracy the interpolated model using Git Re-Basin [1] over different optimization seeds.
Matching and merging n models We now evaluate C2 in matching and merging n models.
The matching is given by the factorized permutations obtained by Algorithm We compare against two baselines: the simple approach of naively averaging the weights without any matching, and the MergeMany approach proposed by Ainsworth al. [1]. The latter is reported in Appendix A for convenience. As reported in Table 1, C3M obtains far superior results in terms of accuracy and loss in all considered settings, with accuracy gains as high as +20%. Moreover, our approach natively yields cycle-cone cycle-consistent permutations: Figure 2b shows that Git Re-Basin [1] accumulates significant error when computing the distance between the source model and the model obtained by applying a cyclic series of permutations, while our approach is able perfectly recover the source model. This further confirmed Figure 2a, where we show the loss and accuracy curves when interpolating between model A and the model mapped back after cyclic permutation.

---

ENNIST
CIPAR10
CTFAR100
Matcher
Accuracy (t)
Loss (J)
Accuracy (t)
Loss (L)
Accuracy (1)
Loss (Q)
train
test
train
test
truin
lest
truin
test
train
test
train
test
Naive
0.03
0.03
3.28
3.28
0.10
0.10
3.07
3.07
0.01
0.01
5,30
5.30
MergeMany
0.88
0.86
1.11
1.13
0.38
0.37
2.08
2,06
Reahet
0.31
3.01
2.76
MergoNany?
0.86
1.13
0.50
0.50
2.34
2.30
0.24
3.31
3.12
C=M²
0.89
0.87
1.07
1.10
0.42
0.40
2.11
2.05
0.34
0.30
2.94
2.63
0.89
0.87
1.07
1.10
0.72
0.69
1.26
1.12
0.53
0.46
2.13
1.67
Naive
0.04
0.04
4.04
4.04
0.10
0.10
2.31
2.31
0.01
0.01
6.22
6.22
MergeMany
Regilet
0.03
0.03
7.17
7.18
VGG16
0,10
0.10
2.36
2.36
Reellet
0,45
0.38
2.32
3.06
MergeMany!
0.03
0.03
4.74
4.72
0.60
0.57
1.43
1.32
0.41
0.35
2.27
2.68
CRM8
0,27
0.27
3.43
3.47
0.11
0.11
2.34
2.34
0.46
0.39
2.25
3.03
CaM
0.60
1.34
0.64
1.23
0.60
1.43
2.23
Table I: Accuracy of the merged model when merging models trained with different initializations.
The best results are highlighted in bold. denotes models after the REPAIR operation.
Models cyclically permuted with Git Re-Basin end up in different basin than the one they started from, while our cycle-consisten. approach ensures that the target model is exactly the same as the source. Wrapping up, approach matches and merges models with significant improvement in performance over the state-of-the-arl, while ensuring cycle-consistent permutations.
Model similarity before and after mapping As we can see in Figure 5, the cosine similarity of the weights of the models 3x higher after mapping the latter the universe. This sug- gests that the initial nuasi-orthogonality of models is at least partially due to neuron permutation symmetries. We also report in Appendix C.1.2 the similarity of the representations between pairs of models. Interestingly, the latter does not change before and after mapping to the universe, but only if we consider similarity measure that is invariant to orthogonal transforma- tions such as CKA [22]. When using measure that does not enjoy this property, such as the Euclidean distance, the repre- sentations become much more similar in the universe space. In short, the models are 3x more similar in the universe space and the mapping affects the representations transformation.
ouppex
(:22
0.3
0.83
0.52
0.12
1
1.01
3.88
0.5L
Maci Spuled
0.20
432
0.12
0.12
6.12
Bofin reapling
Syrdeo
Figure 5: Cosine similarity of the weights of 5 ResNet20 trained on CIFAR10 with 2x width.
orthogonal
Effect activation renormalization Our empirical evidence also points out the benefits of the REPAIR operation [21] that is performed after the merging. In fact, the detrimental effect of model averaging
the activation statistics [21]] still applies when taking the mean of n models instead of two. Our results clearly show the benefit of REPAIR, making it key ingredient of our overall framework. Requiring meaningful interpolation endpoints be effective, REPAIR has lower benefit when em- ployed on the MergeMany algorithm of Ainsworth et al. [1]. In fact, iteratively taking means of different random model subsets and aligning the left-out models to the mean is a more complex process than interpolating between some endpoint models. By taking the mean of models in the universe space, we are instead effectively interpolating between endpoint models that can be used for the computation of the statistics Equation (8). Figure 6 shows the benefit of using the repair operation on 5 VGG models trained on CIF AR10 mapped to the universe space. Specifically we fix one model "a" and we linearly interpolate the universe space with respect to the other models, measuring accuracy before and after applying REPAIR. Other than boosting performance, we observe that the latter reduces the variance over interpolation paths, resulting in the interpolation curves of all the models overlapping. Overall, using the models in the universe as meaningful endpoints to gather activation statistics, our approach can fully leverage activation renormalization techniques such REPAIR.
D-- opdbed
14.1 eqpardd
Figure 6: Interpolation curves of VGG models the universe.
7

---

Lood
1.75
0.65
1.50
0.06 662-
D105
0655
06
0:70-
0.03
010
15
10
Ntraber caddae
Nambee el lankd
Nladon uanta
MarggMazz (taant
C'as" (osta)
Me (tesie) Meegelorg
MargyMare (fna)
(a) MLPs trained over MNIST.
(b) ResNet20 models trained over CIFAR1O.
Figure 7: Accuracy and loss when increasing the number n of models to match and merge.
Increasing n this experiment, we show how the merged model behaves when increasing the number of aggregated models. As we can see in Figure 7a, increasing the number of MLPS up to 20 causes the performance to slightly deteriorate in a relative sense, but remaining stable in an absolute sense it doesn't 't fall below 98%. More surprisingly, Figure 7b shows that for ResNet20 architecture with 4x width the loss and accuracy are not monotonic, but rather they seem to slightly fluctuate. This may hint the merging process being model set, than by its cardinality. Intuitively, model that is difficult to match with the others will induce harder optimization problem, possibly resulting in worse merged model. We dive deeper in the effect of the composition of the set of models in Appendix C.2. In short, our approach is effective in merging a larger number of models, suggesting promise in federated settings.
influenced by the composition of the
Varying widths We now measure how ar- chitectural width affects model merging, tak- ing into consideration ResNet20 architectures with width W E {1, 4, 8, 16). As we can see in Figure 8, width greatly increases the performance of the merged model, reaching the zero-loss barrier first observed in [1] when W = 16. This is in line with the observations relating linear mode connectivity and network widths [11, I]. and confirms the intuition that the merging is only effective when modes can be linearly connected.
Accuney
Loos
2.5
2.1
1.5
5.
15
Witts
Witth
Figure 8: Accuracy and loss when merging 3 ReaNet20s trained over CIFAR10 with different widths. i indicates models after applying REPAIR.
Alternative: fixing one model as universe Alter- natively, could achieve cycle consistency by us- ing one of the source models as reference and learn- ing pairwise maps towards this one. This, however, would require arbitrarily selecting one of the models, making the overall merging dependent trary choice. To see why this matters, we merged 5 Rea by choosing and aggregating the models in its basin. Figure shows severe oscillations in the results, with one model reaching an accuracy as low as 65%, while our approach performs as the best possible reference. This approach, moreover, does not address multi- model merging, it is intrinsically pairwise: multi-task setting, models optimally mapped to ref- erence basin would only be able to solve the task solved by the reference model. This would prevent merging to be used for models containing comple- mentary information, such as knowledge fusion [19]
Anrmary meryel diffeer mélenve
K2-11
$1.36
60
25.0
78.
an arbi-
model as reference
Pairalion FeRAR
Pairsio
Pazeice
Londa
toda
Figure 9: Accuracy of the merged model when mapping towards one arbitrary model (a, b, c, d. e) versus using M33 and the uni- verse space.
8

---

Acssany
Loss
0.8
0.4-
0.0
A-C
A-
B-C
MA-CCC
PIA P
Pa P.
(a) 2D visualization accuracy and loss of the models sampled from the pairwise interpolation lines.
(b) 3D visualization the loss of the models sampled from the pairwise interpolation lines.
Figure 10:
Linear mode connectivity before and after mapping to universe for ReaNet22-22x models trained over CIFAR10 according to Algorithm 1.
or multi-task merging [36]. In our setting, instead, the universe model must by design be a function of all the models and act as a midpoint, hence aggregating information from all the models.
Linear mode connectivity in the universe Figure 10 shows that the loss curves of models interpo- lated the universe are much lower than those interpolated the original space, suggesting that the models are more connected in the former. These results, together with the loss landscape observed in Figure 3, encourage merging the models the universe space due to the lower loss barrier.
5 Related work
Mode Connectivity and model merging. Mode connectivity studies the weights defining local minima. Frankle et [13] studied linear mode connectivity models that were trained for a few epochs from the same initialization and related it to the lottery ticket hypothesis. Without requiring same initialization, Entezari al. [11] speculated that all models share a single basin after having solved for the neuron permutations. Model merging aims at aggregating different models into single one to inherit their capacities without incurring in the cost and burden of ensembling. In this regard, Singh and Jaggi [34] proposed an aptimalltransspen based weight-matching procedure, while Git Re-Basin [1] proposed three matching methods and the MergeMany procedure seen Section Subsequently, REPAIR [21] showed that significant improvement performance of the interpolated model may be obtained by renormalizing its activactions rather than changing matching algorithm. Differently from all these works, consider merging models and propose principled way to perform with cycle-consistency guarantees.
Cycle consistency. Ensuring cycle consistency of pairwise maps is a recurring idea in the computer vision and pattern recognition literature. In the realm of deep learning, earlier studies addressing multi-graph matching achieved cycle consistency by synchronizing ex-post the predicted pairwise permutations [40, 41J. The alternative approach using an object-to-univers. matching framework, which we adopt here, inherently ensures cycle consistency by construction, as demonstrated in [4, 16, 31]. To the best of our knowledge, none of the existing works tackles cycle-consistent alignment of neural models. We refer Appendix A.l for more detailed list of related works.
Conclusions
In this work, we treated the problem of model matching and merging. We first introduced novel weight matching procedure based on the Frank- algorithm that optimizes for the permutation matrices of all layers jointly, and then generalized it to the case of n models. Guaranteeing cycle- consistency, the latter poses principled way to merge set of models without requiring an arbitrary reference point. We then showed the approach to yield superior performance compared to existing ones in merging multiple models in a set of scenarios spanning different architectures and datasets. We believe the formalism to elegantly fit the requirement for the merging operation to unify the different models into cohesive one, rather than mapping all of them to one of the models in the set.

---

Acknowledgments
This work is supported by the ERC grant no.802554 (SPECGEO), PRIN 2020 project no. N0.2020TA3K9N (LEGO.AI), and PNRR MUR project PE0000013-FAIR. Marco Fumero is sup- ported by the MSCA IST-Bridge fellowship which has received funding from the European Union's Horizon 2020 research and innovation program under the Marie Sktodowska-Cume grant agreement No 101034413. We thank Simone Scardapane for the helpful feedback on the paper.
References
[1] Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git Re-Basin: modulo permutation symmetries. In The Eleventh International Conference on Learning Representations, 2022.
Merging models
[2] Federica Arrigoni and Andrea Fusiello. Synchronization problems in computer vision with closed-form solutions. International Journal of Computer Vision, 128, 01 2020. doi: 10.1007 11126-011--1240-xx
[3] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1XXx::6677.064500 2016.
[4] Florian Bernard, Johan Thunberg, Paul Swoboda, and Christian Theobalt. Hippi: Higher- order projected power iterations for scalable multi multi-matching. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), October 2019.
[5] Florian Bernard, Daniel Cremers, and Johan Thunberg. Sparse quadratic optimisation over the stiefel manifold with application to permutation synchronisation. In Neural Information Process- ing Systems, 2021. URL https: //api. www.nti org/ aerg/CorpussD. 238253392.
[6] Daniel Beutel, Taner Topal, Akhil Mathur, Xinchi Qiu, Javier Fernandez-Marques, Yan Gao, Lorenzo Sani, Kwing Hei Li, Titouan Parcollet, Pedro Porto Buarque de Gusmão, et al. Flower: friendly federated learning research framework. arXiv preprint 13X1v:2007.14390, 2020.
[7] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. Emnist: an extension of mnist to handwritten letters, 2017.
[8] Luca Cosmo, Emanuele Rodolà, Andrea Albarelli, Facundo Mémoli, and Daniel Cremers. Consistent partial matching of shape collections via sparse modeling. Computer Graphics Forum, 36(1):209-221) 2017. doi: 10.1111/cgf. 12796.
[9] Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-142, 2012.
[10] Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred A Hamprecht. Essentially no barriers in neural network energy landscape. March 2018.
[11] Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. October 2021.
[12] Marguerite Frank and Philip Wolfe. An algorithm for quadratic programming. Naval Research Logisties Quarterly. 3(1-2):95-110, 1956. doi: https///bb.og//11..0000aaaa:0000
[13] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In Proc. of ICML, volume 119 of Proceedings of Machine Learning Research, pages 3259-3269. PMLR, 2020.
[14] Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew Gordon Wilson. Loss surfaces, mode connectivity, and fast ensembling of DNNs. February 2018.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
10

---

[16] Qi-Xing Huang and Leonidas Guibas. Consistent shape maps via semidefinite programming. In Proceedings of the Eleventh Eunoraphicc/AAMSSCCRR.. Symposium on Geometry Process- ing, SGP 13, page 177-186, Goslar, DEU, 2013. Eurographies Association.
[17] Qixing Huang, Fan Wang, and Leonidas Guibas. Functional map networks for analyzing and exploring large shape collections. ACM Truns. Graph., 33(4), jul 2014. ISSN 0730-0301. doi: 10.1145/201097. 27726111111. URL https: http://oiooorgg///////// 1
2601111
[18] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448 456. pmlr, 2015.
[19] Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. The Eleventh International Conference on Learning Representations (ICLR), December 2022.
[20] Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietre and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models, 2023.
[21] Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari. and Behnam Neyshabur. REPAIR: REnormalizing permuted activations for interpolation repair. The Eleventh International Conference Learning Representations, January 2023.
[22] Simon Korblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. May 2019.
[23] Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009, URL https: wwwp.sssmaaiiscoolaa..com Drrg/CorpuIID: 18268744.
[24] Simon Lacoste-Julien. Convergence rate of frank-wolfe for non-convex objectives. arXiv preprint arXiv: 1607,0033 2016.
[25] Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen. Deep model fusion: A survey, 2023.
[26] Ekdeep Singh Lubana, Eric J Bigelow, Robert P. Dick, David Krueger, and Hidenori Tanaka. Mechanistic mode connectivity. In Proceedings tf 40th International Conference on Machine Learning, volume 202. pages 22965 -30044. PMLR, 23-29 Jul 2023.
[27] Aviv Navon, Aviv Shamsian, Ethan Fetaya, Gal Chechik, Nadav Dym, and Haggai Maron. Equivariant deep weight space alignment, 2023.
[28) Deepti Pachauri, Risi Kondor, and Vikas Singh. Solving the multi-way matching problem by permutation synchronization. C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013. URL httpp:: newrips 2013ffile/3ff http:////.....oooooooooo.663666. Paper .pff.
[29] Fidel A Guerrero Peña, Heitor Rapela Medeiros, Thomas Dubail, Masih Aminbeidokhti, Eric Granger, and Marco Pedersoli. Re-basin via implicit sinkhorn differentiation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20237-20246 2023.
[30] Xingyu Qu and Samuel Horvath. Rethinking model re-basin and linear mode connectivity, 2024. URL https: //arvvv////iv/////eeeeeeeeeeaa org/ 22422 05966.
[31] Frank Schmidt, Eno Töppe. Daniel Cremers, and Yuri Boykov. Intrinsic mean for semi- metrical shape retrieval via graph cuts. In Fred A. Hamprecht, Christoph Schnörr, and Bernd Jähne, editors, Pottern Recognition, pages 446 446,455, Berlin, Heidelherg, 2007. Springer Berlin Heidelberg. ISBN 978-3-540-74936-3
[32] Xinchu Shi, Haibin Ling, Weiming Hu, Junliang Xing, and Yanning Zhang. Tensor power iteration for multi-graph matching. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5062-5070, 2016.
doi: 10. 10.1109/CVPR.2016555)
11

---

[33] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2015.
[34] Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. Hugo Larochelle, Marc aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Ad- vances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[35] Richard Sinkhorn and Paul Knopp. Concerning nonnegative matrices and doubly stochas- tic matrices. Pacific Journal of Mathematics, 21:343-348, 1967. URL https://api semanticacholar wwacchooar.rgg/ooppiiiioo org/CorpusID: :D:
[36] George Stoica, Daniel Bolya, Jakob Brandt Bjomerr Pratik Ramesh, Taylor Hearn, and Judy Hoffman. Zipit! merging models from different tasks without training. In The Twelfth International Conference on Learning Representations.
[37] Paul Swoboda, Dagmar Kainmüller, Ashkan Mokarian, Christian Theobalt, and Florian Bernard.
A convex relaxation for multi-graph matching. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1 1118-11157, 2019. doi: 10. 1019 01141.
[38] Lanhui Wang and Amit Singer. Exact and stable recovery of rotations for robust synchronization. Information and Inference: A Journal of the IMA, 2(2):145-193, 2013.
[39] Lirui Wang, Kaiqing Zhang, Allan Zhou, Max Simchowitz, and Russ Tedrake. Fleet policy learning via weight merging and an application to robotic tool-use, 2023.
[40] Runzhong Wang. Junchi Yan, and Xiaokang Yang. Neural graph matching network: Learn- ing lawler's ' quadratic assignment problem with extension to hypergraph and multiple-graph matching. CoRR, (tbs/1911.. 1. 1308, 2019.
[41] Runzhong Wang, Junchi Yan, and Xiaokang Yang. Graduated assignment for joint multi-graph matching and clustering with application unsupervised graph matching network learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 19908-19999 Curran Associates, Inc., 2020.
[42] Christopher Zach, Manfred Klopschitz, and Marc Pollefeys. Disambiguating visual relations using loop constraints. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 1926-1433, 2010. doi: 10. 1199///P..0000.5539988
[43] Allan Zhou, Kaien Yang. Kaylee Burns, Adriano Cardace, Yiding Jiang, Samuel Sokota, J. Zico Kolter, and Chelsea Finn. Permutation equivariant neural functionals, 2023.
[44] Zhanpeng Zhou, Yongyi Yang, Xiaojiang Yang, Junchi Yan, and Wei Hu. Going beyond linear mode connectivity: The layerwise linear feature connectivity, 2023.
12

---

Contents
Introduction
2 Background
3 Approach
4 Experiments
5
Related work
Conclusions
A
Additional details
14
A.I
Extended related work
14
A.2
Pairwise Frank-Wole Algorithm
14
A.3
MergeMany Algorithm
15
A.4 REPAIR
15
A.5
Convergence and efficiency
15
A.6
Architectural details
16
A.7 Datasets, hyperparameters and hardware details
17
A.8 Proofs
18
B Additional experiments
19
B.1 Pair-wise model matching and merging
B.1.1 ResNet with BatchNorm
................................
19
B.2
Initialization strategies
B.3 Variance of the results in Git Re-Basin
20
B.4 Large-scale matching: ResNet50s trained over ImageNet
20
B.5
Federated Learning
21
C
Additional analysis
22
C.I Similarity of models
................................
22
C.I.1 Representation-level similarity C.1.2 Weight-level similarity
................................ ................................
22
C.2 Merging different subsets
22
Discussion
24
D.I On the cycle-consistency of C3M3
24
D.2 Limitations
24
D.3 Societal impact and broader vision
24
13

---

A
Additional details
Here report in-depth explanations and additional experimental details. In particular, Appendix A. extensively outlines the most related works, Appendix A.2 shows the Frank-Wolft algorithm for the pairwise case, while Appendix A.3 describes the MergeMany procedure presented in [1] for merging multiple models. We also report the REPAIR method in Appendix A.4. Finally, we show how the matching algorithm empirically converges in Appendix A.5.
A.1 Extended related work
We report here a thorough review of works that are relevant to our research, providing a comprehensive understanding of the context of our work.
Linear mode connectivity Mode connectivity is interested in modes, i.e. model parameters at convergence. In this regard, Frankle et al. [13] first studied the connectivity of the parameters of models that were trained for few epochs from the same initialization, while Garipov et al. [14] investigated whether these can be connected through a high-accuracy path without requiring the same initialization. Simultancously, Draxler et al. [10] proposed algorithm to find Minimum Energy Path (MEP) between two modes of a neural network, showing that these paths are mostly flat in both the training and test landscapes. This implies that many minima actually live in shared low-loss valley rather than in distinct basins. On a different perspective, Zhou et al. [43] proposed to study a class of neural functionals which are permutation-cquiiariaa by design. Recent research proposes to study model behavior in the weight space beyond linear mode connectivity: Lubana et al. [26] show that different "mechanisms" in related models prevent simple paths of low loss in the weight space, while Zhou et al. [44] studied the linear connections between the linear features of each layer of differently trained models.
Model merging Model merging [1, 29, 34, 20, 39, 36] has seen a surge of interest in the last years mean to ensemble models without incurring in the added computational cost. One of the first works in this direction is Singh and Jaggi [34], who proposed an optimal-transpe weight-matching procedure. Later, Ainsworth et al. [1] proposed three matching methods, of which being data-free. Closer to our global optimization, Peña et al. [29] proposed a gradient-descent based procedure that iteratively updates soft permutation matrices maintaining their bistochasticity via a differentiable Sinkhorn routine. When the models to match have been trained on different tasks, Stoica al. [36] introduce more general "zip" operation that accounts for features that may be task-specific and further allow obtaining multi-headed models. Most recently, Navon et al. [27] proposed aligning models in the embedding space of a deep weight-space architecture. Finally, weight merging proved useful for large language models [20] and robotics [39]. For complete survey of mode connectivity and model merging, we refer the reader [25].
based
Cycle consistency Cycle consistency is a recurrent idea in computer vision and pattern recognition, where it appears under different names (e.g.. "synchronization" "loop constraints" "multi-way matching") ") depending on the task. In the area multi-view 3D reconstruction, Zach et al. [42] were probably the first to make explicit attempt finding solutions meeting the cycli-conns requirement, although without ensuring theoretical guarantees on the result. In geometry processing, Cosmo et al. [8] ensured cecle-conssstent alignment of collections of 3D shapes using n-fold extension of the Gromov-Wasse Wasserstein distance with sparsity constraints. Overall, cycle consistency is recurring idea in the computer vision [38, 42, graph matching [28, 37, 32] and geometry processing literature [17, 8, 4].
A.2 Pairwise Frank-Wolfe Algorithm
As introduced in Section 3, we optimize a layer- global objective by iteratively optimizing its linear approximation via the the Frank- Wolfe algorithm [12]. We compute the gradient of Equation (1) with respect to each permutation P., as the sum of two contributions for each Vp,: one from permuting the rows of W. and another from permuting the columns of W
( T + (WA1) P++ W
V Vpff =
(6)
from permuting
from permutting columns
14

---

We report in Algorithm 3 the Frank- Wolfe algorithm for the pairwise case.
Algorithm 3 Frank- Wolfe for pairwise Weight Matching
Require: Weights of two models A and B with layers, tolerance Ensure: Approximate solution to Equation (1)
p identity matrices
2: repeat
3: for to do
4: Ph <- permutation acting on rows of W;
5:
1-1 < permutation acting on columns of W; P +- W (
6:
7:
VPL, += (WA)
8:
end for
9:
for Pk E pk do
10:
IT, f LAP(V PM
11: end for
12:
a - LINESEARCH(S, pk II)
for pk E pk do
pkl = PP + 11, end for
14:
15:
16:
until [I](A, B, ph-1)------ ( (,), <e
return p
A.3 MergeMany Algorithm
Algorithm 4 reports the MergeMany procedure originally proposed by Ainsworth al. [1] for merging multiple models, mainly consisting in alternating matching and aggregation until convergence. In practice, each iteration, the procedure picks reference model random and matches all the other models to it. Then, they are all aggregated by averaging the weights.
Algorithm 4 MERGEMANY
Require: Model weights O ON A merged set of parameters O.
Ensure:
1: repeat
2: for E RANDOMPERMUTATIOYL.......
do PEANUTATONCCOODDMMIISSCNTTT!!!! O,)
3: O' + N Eoj 9
4:
5:
+ - (
6:
end for
7:
until convergence
8: return N E 9
A.4 REPAIR
Observing a decay in the variance of the activations of the aggregated model, Jordan et al. [21] proposed REPAIR, which renormalizes the activations of the merged model to match the statistics of the original models. In particular, given two endpoint models with activations X1 and X2, the activations Xa of the interpolated model are renormalized to have statistics:
E [Xa] = (1 a) [X]] a E [X2]
(7)
std (X.) = a) std (X,) + std (X2).
(8)
A.5 Convergence and efficiency
15

---

We report here the convergence of our matching algorithm. In particular, Figure 11 shows the objective values dur- ing the optimization, exhibiting the expected monotonic increase, while Figure 12 shows the step sizes result of the line search at each iteration. Interestingly, Figure 12a shows that the step sizes are generally decreasing, but descend in an alternating manner. This is likely due to the fact that the permutations are obtained as consecutive interpolations, where even steps result in soft permu- tation matrix that the average of the current and next permutation matrix, while odd steps generally result in hard permutation matrix with entries in [0, l]. Figure 13 finally shows the intermediate permutation values during the optimization: at each step, the entries of the permu- tation matrix are the linear interpolation of the current solution and the projected gradient with factor given by the step size. The red values in the figure represent entries currently being updated, which are neither (blue) or O (yellow).
Objective curve
11000
10000
Cbjestive
9000
6000
7000
6000
80
Figure 11: Objective values during the optimization. As guaranteed by the Frank-Wolf algorithm, the objective value increases monotonically.
We report in Appendix A.5 the wall-clock time when merging n = 2, ResNet20 models having 1x, 2x, 4x, 8x and 16x width, together with their number of parameters.
1x
2x
4x
8x
16x
# params
292k
1.166m
4.655m
18. 18.600m
74.360m
n=2
C2M
33.4s
33.5s
40.5s
80.8s
367.8s .8
MergeMany
0.24s
0.4s
3.4s
8.9s
59.4s
n=3
C2M time MergeMany 1.2s
32.9s
83.18s
91.0s 19.5s
162.0s 105.8s
715.8s 892.3s
4.1s
Table 2: Wall-clock time for merging n = 3 ResNet20 models with different widths.
As can be inferred from the table, the scaling laws depend the complexity of the resulting matching problem and cannot be predicted merely from the number parameters, with a 4-fold increase in parameters resulting in no increase in runtime for the first three columns, double increase in the second-last column and a 5-fold increase in the last. Compared to MergeMany, our approach enjoys milder increase in running time when increasing the number of parameters. For simpler settings, however, MergeMany is significantly faster. Being the two approaches on the same order of magnitude and given the one-time nature of model merging, we believe this aspect be of secondary importance, especially considering merging to be, in many cases, an alternative to training a model from scratch.
A.6 Architectural details
We report here the urchitectural details of all the architectures we have used in the experiments.
Multi-Layer Perceptrons We use a simple MLP mapping input to a 256-dimensional space followed by 3 hidden layers of 512, 512 and 256 units respectively, followed by an output layer mapping to the number of classes. We use ReLU activations for all layers except the output layer, where we use log_softmax activation.
ResNet We consider a ResNet20 [15] architecture composed by three ResNet block groups, each containing three residual blocks. The model starts with an initial convolutional layer followed by
16

---

Stop doe 'er codd Broniio
Stop don 've eves inveatiess
1.8
10-
1.5-
Styp Stypron
Step
6.4--
(a) Step sizes for all iterations.
(b) Step sizes for odd iterations.
(c) Step sizes for even iterations.
Figure 12: Step sizes during the optimization.
Initialization
Step 2
Step 3
Step
Step 5
Step 6
Figure 13:
First 6 steps of Algorithm 3 for one permutation matrix. At each step, the new solution is given by the linear interpolation of the current solution and the gradient of Equation (1).
normalization and ReLU activation. then passes through the three block groups with increasing channel sizes (determined by the widen factor) and varying strides, followed by global average pooling and fully connected layer that outputs class logits. As normalization layers, we consider both the most commonly used BatchNorm [18] and, for the sake of comparing with Git Re-Basin, also LayerNorm [3]. The results in the main manuscript are all obtained with LayerNorm, while we report the results with BatchNorm in Appendix B.1.1.
VGG We employ a VGG16 [33] architecture with LayerNorm [3] normalization layers. The model has the following convolutional layer dimensions, with "M" indicating the presence of max-pooling layer
64, 64, M, 128, 128, M, 256, 256, 256 M, 512, 5122,,51,52,,
The convolutional layers are organized followed by a max-pooling layer. The final classifier is composed of three fully connected layers with 512 hidden dimension and ReLU activations.
blocks, each containing 2 or convolutional layers,
A.7 Datasets, hyperparameters and hardware details
We employ the most common datasets for image classification tasks: MNIST [9], CIFAR-10 [23],
EMNIST [7] and CIFAR-100 [23], having 10, 10, 26 and 100 classes respectively. We use the standard train-test splits provided by torchvision for all datasets.
17

---

We use the same hyperparameters as Git Re Re-Basin where possible to ensure a fair comparison. In particular, we train most of our models with batch size of 100 for 250 epochs, using SGD with momentum 0.9, learning rate of 0.1, and weight decay of 10-1. We use cosine annealing learning rate scheduler with warm restart period of 10 epochs and minimum learning rate of O. We report each and every one of the hyperparameters used for each experiment, as well as all the trained models, in a WandB dashboard".
All of the experiments were carried out using consumer hardware, in particular mostly on a 32GiB RAM machine with a 12th Gen Intel(R) Core(TM) i7-12700F processor and an Nvidia RTX 3090 GPU, except for some of the experiments that were carried on a 2080. Our modular and reusable codebase based on PyTorch, leveraging PyTorch Lightnnng to ensure reproducible results and modularity and NN-Template to easily bootstrap the project and enforce best practices.
A.8 Proofs
Theorem A.1. The gradient of the objective function
n n-1
ΣEΣ ((PP)) ((P))
(T T WiPL
p=lo-p-ltst
is Lipschitz continuous, implying our algorithm obtains a stationary point at a rate of O(1/Vl) (24].
Proof. We recall that, for each layer permutation PA {PA, PA...... ...... P of model A, we can define the gradient of our objective function relatively to the model B we are matching towards:
S(PA)
V + Vook Vims =
PA
WAPLL T (W (WAD PA (PE()))) W P
(P (W,") + (W
(P W E+1 PA
To prove Lipschitz continuity, we need to show there exists a constant C' such that V p 1, I.....n, n, l = 1, L.....L [[()))) - f((2)) S CiIPP Qfll- To simplify passages, we only consider a fixed l and perform a generic analysis. We begin by observing that
f(PP) - f(Q%)=
Σ
(PPL (W2) + PP (P(+2) T (P) - Q7)+
(W?)) + (W2+1)
pd E+1 (P WP i+1 (P? !?)
The last form of the above equation can be rewritten as a sum of the two sums:
Σ
(PP + (W2,)) P (P
W (P? - Q9)+
gE(),)|Mp)
(WPPL + (W Pa Σ
T (PP T (P)
!?)
(p))
Since the first term does not depend on either P." or Q². we assume as worst case that its norm O. Then, we remove transposes (since ||M || ||M []] and apply the triangle inequality and the sub-multiplicative property of matrix norms:
[I((P)) S(Q?)) 5
Σ
|PP - -F1
+ ||
|II P P [|
Let C' ma
(1IW3 || + || || || Then, [I/(P)) ((0))) < C
Σ
||P - 0. = C(n
vE(Lm)\Mp)
Link concealed to preserve anonymity. batps://gituub.co//grd aitp///gggoaaannteemettttt
18

---

Matcher
Regget
RogHet
Roglot
189
Text
Naave
2,12
models GIL Be-Beiin Frunk-Rolfe
0A22 0,2
0.19
0.78
0.38
0008
0.02
Table 4: Mean and standard deviation of the test and train loss barrier for each method when matching n models on CIFAR10.
we conclude that f(PP) is Lipschitz continuous for all models and all layers, with Lipschitz constant C'n - 1) depending on both the norm of the weights matrices and the number of models.
B Additional experiments
We report additional experiments and results in this section. In particular, Appendix B.1 presents complete evaluation of our matching method for the pairwise case, showing it to be generally competitive with the state-of-the-an Git de-Basin algorithm [1] and to outperform it on architec- tures employing BatchNorm [18] normalization. We then discuss different permutation initialization strategies in Appendix B.2.
B.1 Pair-wise model matching and merging
As described in Section 3, our formalization can readily be used to match n = 2 models. In this case, the energy is given by Equation (1) and the permutations are not factor- ized. We compare the performance of our approach against the Git Re-Basin algorithm [1] and the naive baseline that aggregates the models by taking unweighted mean on the original model weights without applying any permu- tation. From the data presented in Table 3, observe that the approach is competitive with Git Re-Basin [1], with the two methods exhibiting analogously low test barrier on CIFAR10. Focusing on the ResNet20 architecture, we can see that width plays the same role in both approaches, with the barrier decreasing as it increases. We can also appreciate how, while the same architecture resulted in similar barriers for the two approaches on CIFAR10, the barrier is significantly lower for Frank-Wolf Wolfe in CIFAR100, possibly suggesting that the latter is more robust to the complexity of the dataset.
Barrier
Matcher
Train
Tost
VOCL6 Reaket
Naive
7.00 1.24
8.37 4: 1,23
Git-Rebasin
1.04 = 0.10
1.54 0.13
Prank-Wolfe
0.92 0.06
1.42 0.10
Naive
5.79 - 0.39
7,36 0.38
Sit-Rebasin
0.64
Prank-Wolfe
0.44 0.05
0.63 1 0.06
Table 3: Mean and standard deviation of the test and train loss barriers for each method when matching on CIFAR100.
B.1.1 ResNet with BatchNorm
We also report the results of ResNet20 with 2x width using BatchNorm [18] layers instead of LayerNorm [3] ones. This version, as noted in [21], is in fact harder to match but also the one that is commonly used in practice. We can see in Table that the Frank-Wolfe matcher able to achieve a lower barrier than Git Re-Basin, indicating the approach to be more robust to architectures using different normalization layers.
loss barrier (4)
Matcher
train
test
Naive
4.72 1: 0.86
4.99 1 0.86
Git Re-Basin
4.33 + 0.64
4.62 + 0.65
Frank-Wolfe
3,53 0.58
3.79 1 0.57
Table 5: Mean and stddev of the test and train loss barriers on 2 ResNet20-22 models with BatchNorm normalization.
B.2 Initialization strategies
As introduced Algorithm 1, we initialize each N-dimensional permutation to be the x N identity matrix. We now compare this strategy against two alternatives that provide doubly stochastic
19

---

models
2
3
4
5
6
7
9
mein
stddev
mix gap
Frank-Wolfe
train
0.76
0.78
0.78
0.80
0.77
0.76
0.78
0.75
0.81
0.78
0.018
0.057
0.78
(1.2)
test
0.73
0.75
0.75
0.77
0.74
0.73
0.74
0.72
0.78
0.74
0.018
0.060
0.75
(1.3)
train
0.67
0.69
0.69
0.69
0.62
0.69
0.66
0.71
0.68
0.68
0.023
0.085
0.68
lest
0.64
0.66
0.67
0.65
0.60
0.66
0.63
0.67
0.65
0.65
0.020
0.071
0.65
(2,3)
train
0.75
0.74
0.75
0.72
0.76
0.74
0,70
0.73
0.78
0.74
0.020
0.074
0.76
test
0.70
0.71
0.71
0.68
0.72
0.70
0.67
0.70
0.74
0.70
0.020
0.071
0.72
Table 7: Accuracy of the interpolated model using Git Re- Re-Basin [1] over different pairs of models
(1,2), (1, (2,3) by changing random seed = 1,..., 9 in the weight matching procedure.
models
loes barrier [U]
matrices, i.e., such that their rows and columns sum one: the Sinkhorn initialization [35] that initializes the permutation matrix as the solution of the Sinkhorn-Knopp algorithm [35]; ii) the barycenter of doubly stochastic matrices, i.e. the matrix where each element is given by L/N. Table shows the test barrier of the interpolations of three ResNet20-2x models b, and when using the different strategies over 10 different trials. We can see that the constant initializations (identity and barycenter) work well in general, with the additional benefit of having variance in the results. On the other hand, if computational cost is not a concern, one can still choose to run a pool of trials with different Sinkhorn initializations and finally select the best one, trading this way efficiency with some extra accuracy points.
id
barycenter
Sinkhorn
Gu b)
0.52
0.47
0.60 1: 0.04
0.70
0.64 D 0.06
(0.c)
0.97
0.95
0.92 0.07
Table 6: Test barrier of the interpolations of 3 ReaNet200-2 models using different initializations.
B.3 Variance of the results in Git Re-Basin
As introduced in Section 4, Git Re-Basin [1] depends on a random choice of layers, resulting in variations of up to 10% in accuracy depending on the optimization seed, while our method shows zero variance. While we have already seen the results for model pair Figure 4, we report, for completeness, the results of matching and averaging models with Git Re Re-Basin using different optimization seeds for additional pairs. As can be seen in Table 7, the trend confirmed over these ones, with results significantly oscillating and our approach always above or on par with their mean.
B.4 Large-scale matching: ResNet50s trained over ImageNet
For this experiment, we matched three different ResNet50s trained over ImageNet. We used three publicly available pretrained check- points from timm, namely a1 in1k', c1 1n1k' and ram in1k ". As Table 8 shows. C² 3 underperforms the baseline in this case. To see why, we report in Figure 14 the pairwise accuracies obtained using pairwise weight matching over all the ResNet50 checkpoints available in timm. Let us focus on the triplet (am, a2, ram) and replace the model names with (a, b, c) for clarity. We see that, while the mergings (a, b) and (b, c) result in high-accuracy models, the merging (a, c) yields poor results. Given the cycle consistency method, we inherit the difficulty of the hardest pair, which in this case is (a, c). It is worth noting that this behavior is not present in the other cases investigated in this work, and might be due the considered models being trained with different training schedules and hyperparameters. Future research could investigate new strategies handle such cases, e.g. by iteratively merging models by following max-accuracy path in an accuracy weighted graph.
Matcher
Accuracy
Haiva
0.001
6.91
MergeMany
0.001
6.91
Merge MergeMany'
0.30
4,87
CMM
0.001
6.91
0.07
6.13
Table 8: Accuracy and loss of the interpolated model using different matchers over three ResNet50 models trained on ImageNet.
hhtt:://hhggingffaacccc//mm////e _n1_knkk
20

---

5-
0.50
0/02
0:33
0.00
0.112
01:03
-1.7
2
15
0.90
13
0.07
9.31
6.2%
01:00
01:55
0.00
0:50
0.14
2
0:30
0.1G
065
0.66
0..0
01:31
0.00
0.01
6:17
0:36
0.54
0.00
0.01
0.51
0.01
1313
6.27
0.71
6:30
01:00
0101
0100
0U65
01.64
0.01
0.01
0.20
0.50
1110
0.15
01
61
0.00
11:05
0.00
0000
0.00
0.44
0101
0.67
-12
0.000
25
0:00
0101
11
0.58
100
12k
Martille
Figure 14: Pairwise accuracies obtained using Git Re-Basin [1] over different ReaNet50 models trained on ImageNet. Models are available from the timm library.
Accuracy
Round
10
25
35
45
FedAvg
0.0942
0.394
0.4972
0.5517
0.5699
0.5893
0.6018
0.6063
0.6099
0.6136
C
0.0941
0.4234
0.5193
0.5555
0.5783
0.5978
0.6077
0.6165
0.618
0.622
Table 9: Accuracy over 10 clients in federated learning scenario. We report the accuracy for 50 aggregation rounds, with each client training for 20 local epochs. We report one every five rounds for the sake of clarity.
B.5 Federated Learning
We here report the results of a preliminary experiment where framework in federated learning scenario. To this end, we have used the state-of-the-art federated learning library Flower [6] and employed our matching scheme over set of 10 clients over CIFAR10, each adopting small CNN model. We observe the following:
When all the clients start from the same initialization, our approach has no benefit and falls back to standard averaging. In fact, the optimization process quickly returns identity matrices as permutations, suggesting the models already share the same basin.
When instead we initialize the clients from different random initializations, Tables 9 and 10 show that our approach visibly outperforms FedAVG. particular, the benefits get more pronounced when increasing the number of local epochs. This is in line with the intuition that standard averaging becomes less effective when clients drift due to prolonged local training and too infrequent aggregation.
While these results are not sufficient to claim an overall supremacy of the approach for the task due to the limited evaluation and choice models, they show the approach be promising for the problem and encourage further research.
"https://fflon ai/
21

---

Accuracy
Round
2
3
4
6
7
8
9
10
FodAvg
0.0942
0.2638
0.3543
0.3825
0.4165
0.4505
0.4742
0.4994
0.5169
0.5317
M
0.0947
0.3303
0,3899
0.4441
0.4764
0.4968
0.5184
0.5334
0.5434
0.5536
Table 10: Accuracy over 10 clients in a federated learning scenario. We report the accuracy for 10 aggregation rounds, with each client training for 30 local epochs.
C Additional analysis
In this section, we report additional analyses that complement the results presented in the main text. We first analyze in Appendix C.1 how mapping to universe affects the similarity of the models; then, we evaluate how the composition of the match set affects the accuracy of the merged model in Appendix C.2.
C.1 Similarity of models
We analyze here how similar models before and after being mapped to the universe space, first by comparing their representations and then by comparing their weights.
C.1.1
Representation-Leve similarity
Figures 15a and 15b show the Centered Kernel Alignment (CKA) [22] the representations of ReaNet20 models trained on CIFAR10 with 2x width. The linear version of CKA is defined as
HSIC(X,
CKA(X,Y)
(10)
V /HSIC(X, HSIC(Y,
where HSIC(X,Y) -N-1 (XHX H). H = 11 is centering matrix, and 1 is vector of N ones. The denominator introduced to scale CKA between zero and one, where a value of one indicates equivalent representations. CKA is invariant to orthogonal transformations and isotropic scaling. Being permutations orthogonal transformations, CKA stays exactly the same after mapping the models to the universe. On the contrary, the Euclidean distance of the representations of the models significantly decreases after mapping to the universe, as shown in Figures 15c and 15d.
C.1.2 Weight-level similarity
We have seen in Figure that the cosine similarity of the weights is higher after mapping the weights the universe. This suggests that the models are more similar in the universe, which consistent with the fact that it constitutes a convenient space to merge them. We report here for completeness the Figure 16 the Euclidean distance of the weights of ResNet20 models trained on CIFAR10 with 2x width, showing the same trend as the cosine similarity.
C.2 Merging different subsets
We merge subsets of 5 models from the set of 5 models b, c, d, gauge the effect of the match set composition over the accuracy of the merged model. As shown in Figure 17, we run two different merging schemes: in the former (left column), and then consider subsets only at the aggregation step. In the second analysis (right column), we instead consider model subsets from the start, therefore running the whole matching procedure on the k models before averaging them. This way, we aim to disentangle the error resulting from imperfect matching from the one naturally resulting from the aggregation. We highlight few notable aspects:
globally match all the 5 models jointly
1. While the accuracies are expectedly higher when matching a subset with permutations expressly optimized for that same subset (right column), this not case for n 2, in which the permutations resulting from matching the superset of 5 models yield better results when merging pairs of them. This hints at the added constraint of cycle consistency over wide number of models adding in some cases advisable prior over the search space.
22

---

Centerod Kernel Aligament
Centerod Kernel Aligmment
0.33
0.93
Model Symbol
Model symbol
0.92
0192
0.92
0.92
0.93
0.93
0.92
0.33
0.93
0.92
0.93
0.03
0.92
0.
1393
0.93
0.93
0.92
0.93
(a) Before mapping to universe.
(b) After mapping to universe.
Eaclidean distance between representations
Faclidean distance between representations
281.7
146.4
Niodel bymbol
Model tyymhol
274.1
206.2
146.6
153.1
274.3
268.8
258.4
137.4
(c) Before mapping to universe.
(d) After mapping to universe.
Figure 15: Cented Kernel Alignment and Euclidean distances of the representations of 5 ResNet20 trained on CIFAR10 with width.
Euelidean Distance
Enelidean Distance
80.82
71.03
Moocel Symiool
Mocel byybbol
0
81.42
80.81
71.91
71.72
81.13
$0.98
80.94
71.17
71:24
71.61
81.39
80.77
75
80.97
80.91
71.17
71.34
70.93
71.48
(a) Before mapping to universe.
(b) After mapping to universe.
Figure 16: Euclidean distance of the weights of 5 ResNett0 trained on CIFAR10 with 2x width.
23

---

2. The particular composition of the match set has significant impact over the matching and subsequent merge operation, yielding differences of up 20 accuracy points for the downstream model.
3. The standard deviations before the repair operation (red semi-transparent bars in the plots) are way lower when optimizing for the permutations over the superset of all 5 models; this suggests that the matching difficulty is spread over all the maps jointly, eventually yielding more stable results.
D Discussion
We discuss in this section the limitations of our work, well potential future societal impact.
D.1 On the cycle-consistency of C2M3
Our method is natively cycle- cycle-consistent due to the mathematical formulation of the optimization problem. If we were to not desire cycle consistency, the matching method would fall back to the n = Frank- Wolfe (FW) case presented in Section 3. One would then have to transform the pairwise matching problem n-way matching problem, egg. by using the
FW procedure matching step in the MergeMany [1] algorithm. Results for the n = 2 FW matching are reported in Table 3.
D.2 Limitations
From what we have observed in experiments, permutations satisfying linear mode connectivity of the models are hard to find for most architectures and datasets. In fact, given that there is no practical way to prove or disprove the conjecture for which most models end up in the same basin modulo permutations of the neurons, we cannot be sure that a certain set of models even allows finding such permutations, let alone that the permutations found the community not to rely on the existence of such permutations in general. However, we have also shown that we can always find permutations that improve the resulting aggregated model, which is promising practical result for model merging. As for all the existing works concerning linear mode connectivity and model merging, the resulting models that we obtain are sensible to a wide variety of factors, from training hyperparameters to the optimization algorithm used. Several works have already observed the phenomenon in practice: among these, Ainsworth et al. [1] mention among the known failure modes of their approaches models trained with SGD and too low learning rate, or ADAM coupled with too high learning rate. Jordan et al. [21] show that the chosen normalization layer incredibly affects the accuracy of the resulting merged model, while Qu and Horvath [30] observe learning rate, weight decay, and initialization method play a strong role as well. Being mostly empirical field, most of the technical choices that we make in our work mirror the ones made in previous works and are not based on solid theoretical foundation. We therefore release all our code and encourage the community to investigate further on what training and optimization hyperparameters effect linear mode connectivity and model merging.
the optimal ones. We therefore encourage
D.3 Societal impact and broader vision
The work presented in this paper serves as an additional tool for the community to improve the efficiency of deep learning models. By merging models, we can reduce the computational cost of training and inference, as well as the memory footprint of the models. In fact, by aggregating the information of a set of models into single one with the same architecture, practitioners can benefit of the effects of ensembling without incurring in its computational cost. Moreover, merging is in many cases a practical necessity to guarantee confidentiality and privacy user data, as it allows to train models on different subsets of the data, e.g. originating from different clients, and then merge them to obtain a single model integrating all the information. This is particularly important in the context of federated learning, where the data is distributed among different clients and cannot be shared. We believe that the work presented in this paper can be a stepping stone towards more efficient and deep learning models, and we encourage the community to further investigate the potential of model merging in these contexts.
24

---

654
0.77
0.70
0.
06-
0.54
0L55
0.49
0.:5
64
644
Booda
Bedday
babann
Booda
Woddg
baddy
iaburt
lsbodl
Repacal X000
Warls mome
Reparal
Repaiod
(a) Subsets of 4 out of 5 jointly matched models.
(b) Subsets of matched models out of 5 models.
0.-
0:22
6/7
0/2
0'44
04
0.5
10M
00-
14640
Gaban
Bood
Roso
Bata
Dosoy
(ada)
Buadi
Bosd
boday
Wazk
Va
Yach
Sepiial
Vad
Brakial
(c) Subsets out 5 jointly matched models.
(d) Subsets of matched models out models.
D::7
(L76
0.24
1674
I.74
677
6.7-
012
824
644
011-
Bed
662
w.w Worch 19m0
Regaral ...
100
Vanda
BWw Vaac 11/0
... Reparal 300
Vaxia
Beparal
(e) Subsets of 2 out of 5 jointly matched models.
(f) Subsets of 2 matched models out of models.
Figure 17: Accuracy the resulting model when merging different model subsets. (left) performance of models obtained from aggregating subsets of l: 5 models that were matched jointly. (right) analoguous results for subsets of & models that are instead matched independently. i.e., by only optimizing for the permutations that align those & models and discarding the remaining ones. The semi-transparent bands represent the standard deviation of the accuracy.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
All the claims are supported by experiments and sound reasoning.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations and assumptions of our work Appendix D.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
26

---

Justification:
The paper mostly leverages existing theoretical results and properly cites each of these ones.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We describe in great detail all of the presented algorithms and we report all the technical details for the architectures. We also provide a modular and reusable codebase that respects the highest software engineering standards with configuration parameters provided as separate yaml files. All the experiments were performed by setting reproducible seeds that are logged in WandB.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
27

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide a modular and reusable codebase that respects the highest software engineering standards with configuration parameters provided separate yaml files. Code is highly reproducible and machine agnostic due to the extensive use of frameworks and libraries such PyTorch Lightning, WandB, Hydra and NN-template.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
http://ddde///deeueeeeeee ceabbmisssoonooiiiii ooiicyyyyyy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. htc//ublic/guiiii Sediede///oeeummiiic sssssssoooooooooo onnolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The most important details are provided in the paper, with the yaml configura- tion files in the code providing all the remaining minor details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report means and standard deviations for all experiments when variance is present. The only entries that do not have standard deviation are those that result in a deterministic result that is not affected by random seed.
Guidelines:
The answer NA means that the paper does not include experiments.
28

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: we reported in Appendix the equipment used for the experiments, as well as discussion on the efficiency of our methods.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http///eeuiis///eeeeee cc cc/public/ Caubblcc/Ehiiseee CthicsCuueelines
Answer: [Yes]
Justification:
Our research totally complies with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss in Appendix D the potential positive societal impact of our work.
29

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We adequately cite all of the used datasets and architectures.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
30

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: While we provide codebase that is extensible and modular enough to be reused by many researchers in the field, we not releasing any libraries datasets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
31

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "OCXuYxKCpf", "reviewer": "Reviewer_dfjE", "summary": "The paper introduces Cycle-Consistent Multi-Model Merging for merging neural networks by optimizing neuron permutations globally across all layers, ensuring cycle consistency when merging multiple models. Utilizing the Frank-Wolfe algorithm, this approach addresses inter-layer dependencies and guarantees that cyclic permutations result in the identity map. The method is generalized to handle more than two models by mapping each to a common universe space and thus enhancing alignment robustness.", "questions": "See the weakness part above", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The paper introduces a method for merging neural networks by ensuring cycle consistency through global optimization of neuron permutations, addressing a limitation in existing pairwise approaches\n\n- The use of the Frank-Wolfe algorithm for simultaneous layer optimization and incorporation of activation renormalization demonstrates an innovation and advantages through later experiments", "weaknesses": "- While the paper demonstrates the effectiveness of C2M3 across various (simple) architectures and (toy) datasets, it lacks a detailed analysis of the method's scalability to very large models or datasets. A more comprehensive evaluation of performance and computational requirements for larger-scale applications is desirable to understand its benefits\n\n- Although the method shows promising results in experimental settings, the paper would be strengthened by including case studies or examples of real-world applications where C2M3 has been successfully implemented. This would help demonstrate the practical utility and robustness. Currently it is primarily focused on classification tasks.\n\n- The paper lacks a deep theoretical analysis of the convergence properties and guarantees of the proposed method. Including theoretical insights or proofs regarding the convergence and stability of C2M3 would strengthen the work\n\n- Other minor things:\n    - In abstract the number of models is denoted by $N$, while $n$ is used in the main text\n   - Fig 2 does not have \"Figure 2\" in the caption\n   - the definition 1 is quite ambiguous, stating it as $A\\approx B$. However, it is important to clarify the precise meaning of this approximation in a rigorous manner, especially when defining something clearly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their efforts in providing insightful comments and questions. \n\n**Method’s scalability to large models or datasets.**  \n\n| Paper | Conference | Datasets | Architectures |\n| --- | --- | --- | --- |\n| Git Re-Basin [1] | ICLR22 | MNIST, CIFAR10, CIFAR100, ImageNet | MLP, VGG, ResNet |\n| REPAIR [2] | ICLR23 | MNIST, CIFAR10, CIFAR100, ImageNet | MLP, VGG, ResNet |\n| Deep Weight Space Alignment [3] | ICML24 | MNIST, CIFAR10, STL10  | MLP, VGG, CNN |\n| Re-Basin via Implicit Sinkhorn [4] | CVPR23 | ad hoc polynomial regression dataset, CIFAR10 | MLP, VGG |\n| Model fusion via optimal transport [5] | NeurIPS20 | MNIST, CIFAR10 | MLP, VGG, ResNet |\n| CCA-Merge [6] | ICML24 | CIFAR10, CIFAR100 | VGG, ResNet |\n\nAs can be seen in the table, we are using the **most established set of architectures and datasets considered in all the previous and concurrent literature in the field**. This choice stems from two motivations: 1) for the **sake of comparison** and continuity with respect to previous works; 2) the **complexity of the architecture adds additional challenges** that must be taken into account and requires additional research that is not immediately relevant to the merging approach. In particular, transformer-based architectures require ad hoc mechanisms to handle multi-head attention, positional embeddings and the vast number of residual connections, in fact motivating stand-alone works to do this [8, 9].  We agree, however, with the reviewer that we are lacking a very large-scale case and, therefore, are also training ResNet50 endpoints on ImageNet. We aim to include these results as soon as they are available. Given our limited computational budget, these may be ready for the camera-ready. \n\n**Real-world applications**. We thank the reviewer for the suggestion. Tackling the problem of model merging, we inherit all the applicative domains suggested in relevant prior works, such as federated learning [1, 3], incremental learning [4], and continual learning [10]. While its foundational nature, in our opinion, makes examples of real-world applications out of scope for our paper, we share the curiosity of the reviewer in assessing its effectiveness in such a scenario. Therefore, we are currently setting up a federated learning experiment whose results we hope to share during the discussion phase.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pHMHSkl0xk", "reviewer": "Reviewer_TBfo", "summary": "This paper further put forward a kind of cycle consistent Multi-Model Merging to merge models after permute simultaneously. It addresses the limitations in previous approaches that only handled pairwise merging. It uses a \"universe\" space to factorize permutations between models, optimizing all layer permutations simultaneously using the Frank-Wolfe algorithm.", "questions": "- You mentioned: \"the resulting models that we obtain are sensible to a wide variety of factors, from training hyperparameters to the optimization algorithm used\" in the paper. Could you please elaberate more on this?\n- What is your convergence speed comparing to git-rebasin (MergeMany). It would be great to see a Pareto Front showing that given a specific size of set of models, the trade-off between computation time and accuracy and see which algorithm occupies more Pareto Optimal solutions\n- Is that possible to do some exps on CLIP-based/LLM (like the phi-3 which only has 3B parameters) larger networks? There are bunch of CLIP-fine-tuned networks with the same architecture and LLMs fine-tuned based on phi-3 etc. The exps are done only on EMNIST, CIFAR10 and CIFAR100. The largest network being used is ResNet-16/VGG-16 which are a bit out-of-data nowadays. (I know Git-rebasin was working on VGG-16 and ResNet 20 but still...)\n- In figure 1.b, can you also calculate the cosine similarity? After all, L2 distance does not make much sense in high dimension space.\n- The eq 3 is confusing, I am not sure if the author wants to sum p from 1 to n while not equals to q or sum (p,q) pairs from (1,1), (1,2), ..., (n-1,n) while p!=q. I am asking this because the author mentioned \"In order to generalize to n models, we jointly consider all **pairwise**\nproblems\". But the equation 3 does not show the \"pairwise\".\n- What is the meaning of color in figure 3?", "limitations": "Yes, the author have mentioned the limitations well in the appendix.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Deterministic result: independent of the random choice of layers\n- The \"universe\" space to factorize permutations between models solves issues with previous methods that could accumulate errors when applying cyclic permutations.\n- This paper has sufficient analyze and discussion on the model width, number of merged models, and linear mode connectivity in the universal basin.\n- The algorithm to factorize is data-free (based on the Frank-Wolfe algorithm)", "weaknesses": "- Sensitive to hyper-parameters\n- Lack of theoretical guarantee (I personally think there is not much researchers can do in this model-merging sub-domain. Especially in large networks)\n- Exps are down on small datasets and small networks. Could do some exps on larger network and larger dataset\n- What is the convergence speed given a different number of parameters? What about comparing to other algorithm such as MergeMany?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments and spot-on questions. We will now do our best to address each and every critique and question.\n\n**Sensitivity to hyperparameters**\n\nWe thank the reviewer for raising this point. We would like, in fact, to clarify this aspect: **when, in the limitations, we refer to merging methods being sensible to hyperparameters, we do not refer to this particular approach**, but to the whole linear mode connectivity phenomenon. Different works have, in fact, observed this in practice: among these, Ainsworth et al. [1] when listing the known failure modes of their approaches, mention models trained with SGD and too low learning rate, or ADAM coupled with too high learning rate. Keller et al. [2] show that the chosen normalization layer incredibly affects the accuracy of the resulting merged model, while Qu et al. [3] observe learning rate, weight decay, and initialization method to play a strong role as well. We therefore argue that the sensitivity to hyperparameters is not a peculiarity of our approach, but something we observed throughout all the approaches we used and that has been confirmed in the existing literature. We added this clarification to the main manuscript.\n\n**Lack of theoretical guarantees**\n\nWe thank the reviewer for the comment. As explicitly stated in the limitations of the paper, we share with the reviewer an overarching wish for an improved theoretical ground for model merging and linear mode connectivity. This is mostly due to the whole permutation-based model merging field building upon a conjecture that cannot be proven or disproven due to the huge number of possible neuron permutations. We would like, however, to remark that **this is not something that has to do with our research in particular but with the field in general** and that, differently from several works in the field, we provide a **principled approach that indeed holds some guarantees**. While this may be just a brick on a bumpy road, this brick is stable and allows further work to build upon it.\n\n**Small datasets and networks**\n\nWe thank the reviewer for the suggestion. \n\n| Paper | Conference | Datasets | Architectures |\n| --- | --- | --- | --- |\n| Git Re-Basin [1] | ICLR22 | MNIST, CIFAR10, CIFAR100, ImageNet | MLP, VGG, ResNet |\n| REPAIR [2] | ICLR23 | MNIST, CIFAR10, CIFAR100, ImageNet | MLP, VGG, ResNet |\n| Deep Weight Space Alignment [3] | ICML24 | MNIST, CIFAR10, STL10  | MLP, VGG, CNN |\n| Re-Basin via Implicit Sinkhorn [4] | CVPR23 | ad hoc polynomial regression dataset, CIFAR10 | MLP, VGG |\n| Model fusion via optimal transport [5] | NeurIPS20 | MNIST, CIFAR10 | MLP, VGG, ResNet |\n| CCA-Merge [6] | ICML24 | CIFAR10, CIFAR100 | VGG, ResNet |\n\nAs can be seen in the table, we are using the **most established set of architectures and datasets considered in all the previous and concurrent literature** in the field. This choice stems from two motivations: 1) for the **sake of comparison** and continuity with respect to previous works; 2) the **complexity of the architecture adds additional challenges** that must be taken into account and requires additional research that is not immediately relevant to the merging approach. In particular, the architecture suggested by the reviewer is a transformer-based architecture and requires ad hoc mechanisms to handle multi-head attention, positional embeddings, and the vast number of residual connections, in fact motivating stand-alone works to do this [4, 5].  We agree, however, with the reviewer that we are lacking a very large-scale case and, therefore, are also training ResNet50 endpoints on ImageNet. We aim to include these results as soon as they are available. Given our limited computational budget, these may be ready before the end of the discussion phase or for the camera-ready. \n\n**Convergence speed**\n\nWe thank the reviewer for the question. We report here the wall-clock time when merging n=2,3 ResNet20 models having 1x, 2x, 4x, 8x, and 16x width, together with their number of parameters.  \n\n|  | 1x | 2x | 4x | 8x | 16x |\n| --- | --- | --- | --- | --- | --- |\n| # params | 292k | 1.166m | 4.655m | 18.600m | 74.360m |\n| C2M3 time n=2 | 33.4s | 33.5s | 40.5s | 80.8s | 367.8s |\n| C2M3 time n=3 | 32.9s | 83.18s | 91.0s | 162.0s | 715.8s |\n| MergeMany time n=2 | 0.24s | 0.4s | 3.4s | 8.9s | 59.4s |\n| MergeMany time n=3 | 1.2s | 4.1s | 19.5s | 105.8s | 892.3s |\n\nAs can be inferred from the table, the scaling laws depend on the complexity of the resulting matching problem and cannot be predicted merely from the number of parameters, with a 4-fold increase in parameters resulting in no increase in runtime for the first three columns, a double increase in the second-last column and a 5-fold increase in the last. Compared to MergeMany, our approach enjoys a milder increase in running time when increasing the number of parameters. **We also included a rigorous proof determining the convergence rate of the algorithm in the rebuttal document**. \n\n**Cosine similarity of the models**\n\nWe thank the reviewer for the suggestion. We report here the cosine similarity between a model and the model obtained by cyclically applying the permutations obtained with git re-basin and C2M3 respectively. \n\n|  | (a, b, c, a) | (b, c, a, b) | (c, b, a, c) | (a, c, b, a)  |\n| --- | --- | --- | --- | --- |\n| git re-basin | 0.251 | 0.251 | 0.251 | 0.251 |\n| C^2M^3 | 1 | 1 | 1 | 1 |\n\nThe result is analogous to that observed in the manuscript with the L2 distance, showing the lack of error accumulation in the proposed approach.\n\n**Ambiguity in equation 3**\n\nWe thank the reviewer for pointing out this ambiguity. We confirm their understanding of the equation summing over all pairs of models `(1, 1), (1, 2), …, (n-1, n)` . For the sake of clarity, we replaced the equation in the manuscript with a triple sum, for each model `p=1, ..., n-1`, for each model `q=p+1, ..., n` and each layer `l=1, ..., L`.\n\n(TBC)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8A15fb1wYG", "reviewer": "Reviewer_mu2n", "summary": "The paper proposes a new, cycle-consistent method of merging more than two neural networks in weight space by simultaneously solving multiple permutation-based merge process. The key innovation is to ensure cycle consistency when merging $n>2$ models. The authors showed that the method, together with techniques like REPAIR, can lead to substantial benefits across various architectures and datasets.", "questions": "1. The major concern I have is related to the global optimization process itself for model merging. How \"easy\" is this optimization process and why conditional GD? How does the optimization scale with wider/deeper networks? I suspect that the empirical observation in Fig. 12(a) in Appendix A.5 exactly suggests the instability in the optimization algorithm. \n\n2. The authors reported the walk-clock time efficiency for merging 2 models in Appendix A.5. How slow is the method for merging $n>2$ models (e.g., say 5, 10), especially compared to setting one model as the \"universe space\" and perform $n-1$ pairwise optimizations?\n\n3. How exactly is the merging factor $\\alpha$ determined when trying to use REPAIR for multiple models in Table 1?\n\n4. While by design the transformation to/from the universe weight space is invertible, but in practice depending on the quality of the optimization problem solving, the actual $P^{AC} \\circ P^{CB} \\circ P^{BA}$ could in theory still be not so close to the identity matrix. Maybe I missed something, so it'd be great if the authors can clarify.", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a novel alignment algorithm that, by construction, ensures cycle consistency for merging multiple models. The authors specifically showed the benefit of merging models in a universe space $u$ as a bridge for model permutations. I personally found the idea of a universe space neat and sensible, and this method could be a nice advance in the field of model interpolation & merging.\n\n2. Relatively extensive comparison with existing best methods and across different datasets and settings (including some analysis on the loss landscape, etc.)\n\n3. The paper demonstrated compatibility of the method to adopt techniques like REPAIR which helps maintain good performance (i.e., practical enhancement over other prior methods)", "weaknesses": "1. While the concept of a universe weight space is nice, the presumption of neuron permutations and the related theoretical foundations are still based strongly on prior work.\n\n2. I found the paper fail to analyze the pros and cons of the simultaneous global optimization in the main paper (though there are some in the appendix). Specifically, the Frank-Wolfe process will involve a $O(n^2L)$ loop which would be very costly compared to the more \"local\" approach by Ainsworth et al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments and spot-on questions. We now proceed to address the raised concerns and answer the questions to the best of our capabilities.\n\n- **W1: model merging foundations based on prior work.** We agree with the reviewer that our work builds on top of existing work and partially leverages the same foundations. This is true, however, for each permutation-based model merging work, e.g. [1, 2, 3, 4, 5, 6], and we believe that sharing a set of common assumptions is inevitable in a subfield. We would also like to remark that **none of the existing works provably guarantees cycle consistency**, with the problem itself being overlooked. Therefore, while we acknowledge owing the existence of this research to prior work that has laid the foundations of the field, we argue that our work takes it a step further and, being theoretically grounded, has the potential to be itself a foundation for future research.\n- **W2: pros and cons of global vs local optimization.** We thank the reviewer for bringing this up. The answer is double-fold: i) **global optimization is required to enforce cycle consistency**, as it is an inherently global property; ii) **global optimization removes the arbitrariness in the layer iteration order**, which, as we report in Figure 4 and Table 6, results in a marked variance in the results.  \nMore in detail, it is not clear how cycle consistency, a global property of the model-level transformation, can be achieved using layer-local optimization. For this reason, we do not only choose global optimization due to the deterministic nature of its results but also because it is the only way to achieve our main and foremost objective — that of having cycle-consistent permutations between models. Agreeing with the reviewer about the importance of these considerations, we intend to use the extra page in the camera-ready version to clarify them.\n- **Q1: why conditional gradient descent.** We thank the reviewer for the question. Conditional gradient descent (Frank-Wolfe) is a particularly suitable algorithm for the problem we are trying to solve as it **only requires computing the gradient**, for which we derived a closed form, and enjoys two advisable properties: i) **monotonicity of the objective function** (see also Figure 11 in appendix) ii) **guaranteed convergence rates**, for which we added a proof in the appendix. Regarding the difficulty of the process, the monotonicity of the loss makes it arguably stable, even if the step size is decreasing in an alternating manner. As briefly explained in Appendix A. 5, we believe this behavior to reflect a fixed pattern in the optimization rather than an instability in the overall process. Regarding the scalability of the approach with larger networks, we report here its wall-clock time when merging n=2, 3 ResNet20 models having 1x, 2x, 4x, 8x, and 16x width, together with their number of parameters.\n    \n    \n    |  | 1x | 2x | 4x | 8x | 16x |\n    | --- | --- | --- | --- | --- | --- |\n    | # params | 292k | 1.166m | 4.655m | 18.600m | 74.360m |\n    | C2M3 time n=2 | 33.4s | 33.5s | 40.5s | 80.8s | 367.8s |\n    | C2M3 time n=3 | 32.9s | 83.18s | 91.0s | 162.0s | 715.8s |\n    | MergeMany time n=2 | 0.24s | 0.4s | 3.4s | 8.9s | 59.4s |\n    | MergeMany time n=3 | 1.2s | 4.1s | 19.5s | 105.8s | 892.3s |\n    \n    As can be inferred from the table, the scaling laws depend on the complexity of the resulting matching problem and cannot be predicted merely from the number of parameters, with a 4-fold increase in parameters resulting in no increase in runtime for the first three columns, a double increase in the second-last column and a 5-fold increase in the last.   Compared to MergeMany, our approach enjoys a milder increase in running time when increasing the number of parameters. **We also included a rigorous proof determining the convergence rate of the algorithm in the rebuttal document**. \n    \n- **Q2: Efficiency of the method for n>2.** We thank the reviewer for the question. To answer it, we computed the matching and merging time when merging $n=2, \\dots, 10$ ResNet20 models with 4x width, as can be seen in Figure 1 of the rebuttal document. Compared to the pairwise baseline that maps all the models to a fixed one, our approach incurs a significantly steeper cost. However, Figure 2 (rebuttal document) shows that the latter also suffers from a performance decrease when increasing $n$ which is much more pronounced, making it not advisable for the task.\n- **Q3: Merging factor alpha.** We thank the reviewer for pointing out this missing detail. In all our experiments, we have used $\\alpha = \\frac{1}{n}$, where $n$ is the number of models to merge. This is indeed usually the hardest case, as it defines a point that is the furthest from the individual basins.\n- **Q4: cycle consistency.** We thank the reviewer for allowing us to clarify this aspect. Luckily, cycle consistency, i.e., the property for which applying a cycle of such transformations leads back to the starting point, holds in this case as long as each transformation is orthogonal. Looking at Figure 1 in the manuscript, it can be appreciated how mapping $A$ to another model $B$ means always passing through $U$  and then to $B$ with $P_B$, and that mapping $B$ to another model $C$ means passing through $U$ with $P_B^T$. This means that, as long as $P_B P_B^T = I$, the effect of mapping $A$ to $B$ is nullified. Since the proposed algorithm returns permutations (a subspace of orthogonal transformations), cycle consistency always holds for the resulting permutations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their insightful comments and questions. We are happy to see the contribution of our work be appreciated, with reviewers finding the work innovative, empirically advantageous, and addressing a limitation in existing approaches (djfE). The concept of a universe space was found to be neat and sensible, and a nice advancement to the field (mu2n). Moreover, we are happy to see the benefits of the approach acknowledged, being deterministic, data-free and avoiding the accumulation of error when permuting cyclically (TBfo). Finally, we are glad to see that the experiments and comparisons with the best existing methods were found to be extensive (mu2n) and the analysis and discussions to be sufficient (TBfo). 
Having done our best to address all the raised concerns, we remain available for any further clarification or doubt.

Meta-review: The paper identifies a limitation in the existing model merging methods and proposes adding a constraint of cycle consistency through their optimization formulation. The merging of the models is done through global optimization and the authors propose a Frank-Wolfe algorithm for the same. The solution is further improved through layer-renormalization. All the reviews agree on benefits of the cycle consistency constraint for model merging as well as on the novelty of proposed Frank-Wolfe algorithm. The strengths of this paper lies in the novel constraint, the proposed algorithm and its benefit for model fusion. 

The main concerns that have been raised in the reviews are on

1. Experimentation only done with smaller datasets & networks,
2. Convergence properties of the proposed algorithms,
3. Real world case-study which demonstrates practical use of the model merging methods.

The rebuttal from the authors address some of these concerns - 
1. The authors claim that they are using the same datasets as Git-rebasin (prior-works) to benchmark, as well as (currently) training ResNet50 endpoints on ImageNet;
2. For convergence they have included additional proofs as well as shared a table clocking the model merging;
3. For the real world case study they share a Fed-learning example which seems a simpler example.

I recommend *acceptance of this paper with minor revisions*. Following are the specifics that should be addressed in the final version -
1. Please include additional results using ResNet50 and ImageNet.
2. Please include convergence speed tables as well as proofs at appropriate places.
3. Please include discussions on results for Fed-learning highlighting the cases when the method works and the cases 
4. A discussion or ablation experiment which considers only the FW algorithm without the consistency constraint.
Decision: Accept (poster)