Linear Mode Connectivity between Multiple Models modulo Permutation Symmetries
Authors: Akira Ito
Masanori Yamada
Atsutoshi Kumagai
Venue: ICML 2025 poster (2025)
Keywords: Linear mode connectivity
deep learning
permutation symmetry
Primary area: deep_learning->everything_else
TL;DR: We propose a permutation search method using a straight-through estimator for merging multiple neural network models.

Abstract: Ainsworth et al. empirically demonstrated that linear mode connectivity (LMC) can be achieved between two independently trained neural networks (NNs) by applying an appropriate parameter permutation. LMC is satisfied if a linear path with non-increasing test loss exists between the models, suggesting that NNs trained with stochastic gradient descent (SGD) converge to a single approximately convex low-loss basin under permutation symmetries. However, Ainsworth et al. verified LMC for two models and provided only limited discussion on its extension to multiple models. In this paper, we conduct a more detailed empirical analysis. First, we show that existing permutation search methods designed for two models can fail to transfer multiple models into the same convex low-loss basin. Next, we propose a permutation search method using a straight-through estimator for multiple models (STE-MM). We then experimentally demonstrate that even when multiple models are given, the test loss of the merged model remains nearly the same as the losses of the original models when using STE-MM, and the loss barriers between all permuted model pairs are also small. Additionally, from the perspective of the trace of the Hessian matrix, we show that the loss sharpness around the merged model decreases as the number of models increases with STE-MM, indicating that LMC for multiple models is more likely to hold. The source code implementing our method is available at https://github.com/e5-a/STE-MM.

Linear Mode Connectivity between Multiple Models modulo Permutation Symmetries
Akira Ito
Masanori Yamada
Atsutoshi Kumagai 21
Abstract
1. Introduction
Deep neural networks have achieved great success in vari- ous fields, including image classification, speech recogni- tion, and natural language processing (Vaswani et al., 2017; van den Oord et al., 2016; Zhao et al., 2023). The opti- mization of the large neural network (NN) models used in these applications poses a massive non-convex optimization problem. Nevertheless, stochastic gradient descent (SGD), despite its simplicity, consistently finds good solutions. One hypothesis for this seemingly contradictory situation is that the loss functions of NNs have much simpler structure than we might imagine. Several prior studies have empirically shown that solutions of different NNs can be connected by simple low-loss nonlinear paths. Furthermore, in recent years, Entezari al. (2022) hypothesized that the following conjecture holds when all permutation symmetries of NNs are taken into account.
Ainsworth et al. (2023) empirically demonstrated that linear mode connectivity (LMC) can be achieved between two independently trained neu- ral networks (NNs) by applying an appropriate pa- rameter permutation. LMC is satisfied if linear path with non-increasing test loss exists between the models, suggesting that NNs trained with stochastic gradient descent (SGD) converge to single approximately convex low-loss basin under permutation symmetries. However, Ainsworth et al. (2023) verified LMC for two models and provided only limited discussion on its extension to multiple models. In this paper, we conduct more detailed empirical analysis. First, show that existing permutation search methods de- signed for two models can fail to transfer multiple models into the same convex low-loss basin. Next, we propose a permutation search method using straight-through estimator for multiple models (STE-MM). We then experimentally demonstrate that even when multiple models are given, the test loss of the merged model remains nearly the same as the losses of the original models when using STE-MM, and the loss barriers between all per- muted model pairs are also small. Additionally, from the perspective of the trace of the Hessian matrix, we show that the loss sharpness around the merged model decreases the number of mod- els increases with for multiple models is more likely to hold. The source code implementing our method is available at https::///. / thub. com/ ham/e5-a e5 a //STE-MM...
Conjecture 1.1 (Permutation invariance, informal). Let O, and 0 be two SGD solutions (trained model parameters). With high probability, there exists a permutation such that the loss barrier between O₂ and π (0,) is sufficiently small.
Here, barrier refers to the increase in the loss observed during linear interpolation between two models. When the barrier between two models is sufficiently small, we say that linear mode connectivity (LMC) holds between those models. Entezari et al. (2022) hypothesized that LMC can achieved by considering the permutation symmetries of NNs. Subsequently. Ainsworth al. (2023) and Singh & Jaggi (2020) empirically demonstrated that appropriate permutation search methods can identify permutations that satisfy Conjecture 1.1. Based on these findings, Ainsworth et al. (2023); Entezari et al. (2022) argued that the weights of trained multiple models can be transferred into a single approximately convex loss basin. this claim holds, the optimization of NNs could be interpreted as approximately convex optimization, thereby explaining the effectiveness of SGD. In addition, analyzing LMC among multiple mod- els under permutation symmetries is important not only for understanding how SGD works but also for model merging, where independently trained models are combined. For ex- ample, permutation based model merging has the potential to combine multiple models without significant costs, such
indicating that LMC
'NTT Social Informatics Laboratories "NTT Computer and Data Science Laboratories. Correspondence to: Akira <akirr. nnt.ttmmt......
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Linear Mode Connectivity between Multiple Models modulo Permutation Symmetries
as training. Indeed, several prior studies (Singh & Jaggi, 2020; Wang al., 2020; Peña al., 2023; Yamada al., 2024) have proposed methods for model merging, federated learning, and continual learning based on the permutation symmetries of NNs.
2. Background
2.1. Notation
For any k E N. let [k] = (1, Bold uppercase and lowercase variables represent matrices (e.g.. X), and vectors (e.g., æ), respectively. For any matrix X, vec(X) denotes its vectorization, and ||X denotes its Frobenius (L²) norm.
In this paper, we investigate more deeply whether the weights of multiple models can be transferred into a sin- gle (approximately convex) loss basin. First, we show that gathering multiple models into single loss basin is difficult by simply applying a permutation search method that targets only two models. We also demonstrate that, although some previous studies (Ainsworth et al., 2023; Crisostomi et al., 2024) have proposed permutation search methods for mul- tiple models, the test loss and the accuracy of the merged model deteriorate the number of models increases in these methods. To overcome this issue, we propose novel permu- tation search method using straight-through estimator for multiple models (STE-MM), and experimentally confirm its effectiveness.
2.2. Permutation Invariance
For simplicity, consider L-layer multilayer perceptrons (MLPs) f(x; 0) while our analyses are applicable any model architecture. Here, E I is an input to the NN, and 0 E Rdpramm represents the model parameters. Regard- ing the f-th layer output 2g, we have 20 = æ. and, for all E [L]. 21 = o (W + by). where o denotes the ac- tivation function, and We and be denote the L-th layer's weight and bias, respectively. Note that, in this MIP, the model parameters are given by = || (vec Wx) || by), where || represents the concatenation of vectors. Neural networks (NNs) have permutation symmetries within their weight space. Let us consider an NN with model param- eters 0. For the f-th layer, the following equation holds: = P Pz = P a (PW Pb;). where P repre- sents a permutation matrix. Since permutation matrices are orthogonal, by permuting the input to the (f 1-st layer us- ing PT, the model's 'ss parameters can be transformed without affecting the NN's input-output functionality. Specifically, the updated weights and bias are expressed as W PWe b/ = Pbr, and W e+1 We+1 This type of permuta- tion can be applied independently to each layer. We define the tuple of permutation matrices applied across all layers as π = (Pt) Furthermore, for a given model 6, applying a permutation я to is denoted as T
Contributions. This paper makes the following three con- tributions:
Difficulty in satisfying LMC among three more models. We demonstrate that simply applying conven- tional permutation search methods, which focus only on two models, makes it difficult to establish LMC among multiple models. In addition, we show that even when using existing permutation search methods for multiple models (Ainsworth et al., 2023; Crisos- tomi et al., 2024), the performance decreases when the multiple models are merged.
Matching methods for multiple models. We propose a permutation search method using a treiigh-tthroug estimator for multiple models (STE-MM). STE-MM, permutations are explored by repeatedly solving the linear assignment problem (LAP), which incurs a high computational cost. To address this, we also propose method to accelerate the LAP solver.
2.3. Permutation Selection For Two Models
Ainsworth et al. (2023) introduced two methods, weight matching (WM) and the straight-through estimator (STE), to identify permutations which satisfy Conjecture I.1.
Weight matching. The WM aims find a permutation क that minimizes the L2 distance between two models!:
Effectiveness verification of STE-MM through ex- periments and loss sharpness analysis. We evaluate STE-MM on MLP, VGG- 1, and ResNet-20 trained on MNIST, FMNIST, and CIFAR-10. 10. Our results show that STE-MM enables the merging of multiple models while maintaining test accuracy and loss comparable to the originals. Additionally, the merged models ex- hibit reduced loss sharpness, which decreases further as more models are merged. These findings suggest that STE-MM facilitates the transfer of multiple mod- els into an approximately convex loss basin.
02 T (0,))1 = Σ Iw - P.W.! (b) P_I.? (I)
RE(L]
where, without loss of generality, Pz = I and P. = I,
with I denoting the identity matrix. Ainsworth et al. (2023) propose a method to approximately solve this optimization by iteratively solving the linear assignment problem (LAP)
because this optimization problem is NP-hard (Koopmans &
Although the focus is on weights, biases also be incorpo- rated by concatenating them with the weights.
2

---

Linear Mode Connectivity between Multiple Models modulo Permutation Symmetries
Beckmann, 1957; Sahni & Gonzalez, 1976; Ainsworth et al., 2023). We will describe their algorithm in Section 4.2.
Equation (4) shows that adding or subtracting a constant to any row or column of the cost matrix C does not change the optimal solution (permutation matrix) of the primal problem. This property will be utilized in Section 4.2.
Straight-througe estimator. STE directly minimizes the loss barrier between the two models. Its objective function is expressed arg min- & ( (0. + π (06))) Since solving this directly is difficult, Ainsworth et al. (2023) proposed a strategy that splits the optimization into two stages: a forward pass to determine permutations and backward pass to train the model using the obtained permutations. By iteratively repeating these steps, they approximated the solution effectively.
3. Difficulty in Transferring Multiple Models into Convex Loss Basin
Ainsworth et al. (2023); Entezari et al. (2022) have claimed that SGD solutions can be transferred into a single approxi- mately convex loss basin by identifying a permutation that satisfies LMC between two SGD solutions. They experi- mentally demonstrated the existence of such permutation when the model width is sufficiently large for MLP, VGG, and ResNet models trained on MNIST and CIFARIO.
2.4. Linear Assignment Problem
The linear assignment problem (LAP) is used to determine the permutation matrix in permutation search methods such as WM and STE. Given cost matrix C' and a permutation matrix P. the LAP can be formulated as follows:
However, even if this experimental observation holds gener- ally, it does not necessarily imply that SGD solutions can be transferred into single convex basin. For instance, consider three SGD solutions 0.. Os, and Oz. Assume that there exist permutations such that LMC holds for every model pair. In this case, there exist permutations Th and Te such that LMC holds between 0₂ and Tu((0) (0s), as well as between O, and Te (Oa)). If these parameters can be transferred into the same convex basin, the loss within the triangle formed by O.. Ty To(O)) (0)), and Te())) must be uniformly low. However, there is no guarantee that ne (t,) and (+.))) are linearly mode- connected. Note that, from the assumption, there should (2)
minimize
1.
subject to
Σ Pt. = 1, E [m],
Σ Po. 1, Vi [m].
P.j 2 0, Vi,j [m],
exist permutation иT such that LMC holds between Ty (T,(s) and π (T))).)). but this does not necessarily mean LMC holds between (T,))) and (0.).. Therefore, it remains un- clear whether these parameters can be gathered into single convex basin. In this section, we experimentally demon- strate that as the number of models increases, the loss of the merged model increases, revealing that matching algorithms only for models are insufficient.
where m denotes the size of the matrices C and P. and Cr and P.,, represent the jith) components of matrices C' and P. respectively.
Classical algorithms such as the Hungarian algorithm and the Jonker- Volgenant algorithm (Kuhn, 1955; Jonker & Vol- genant, 1987) reformulate the problem into the following dual problem indirectly obtain the optimal solution:
maximize
Σ Cu++3 Σn,, ug:
3.1. Experimental Results
We experimentally demonstrate that matching algorithms between two models are insufficient for gathering multi- ple SGD solutions into single convex basin. Specifically, we first prepare SGD solutions 0,, ......,................... O,. such that
subject to
U. + uj S Pis,Vi,y E [m],
(3)
where u E RD and v E R are dual variables.
Next, for
all i E [n - 1]. we identify a permutation LMC holds between On and (O.). Then, we investi- gate the test loss and accuracy of the merged model of n models 0I...., 0n-1 after applying the permutations, Eie(x T. (O.). If the n SGD solutions can be trans- ferred into single convex basin using the permutations II, ........ nx it is expected that the test loss and accu- racy of these merged models would remain unchanged when the number of models used for model merging increases.
According to the complementary slackness theorem of linear programming, the necessary and sufficient condition for the feasible solutions of the primal and dual problems to be optimal solutions is given by:
P.,( - vs - uz) = 0, Ji.j E [m].
(4)
From this complementary condition, if the optimal solutions o" and u of the dual problem Equation (3) are obtained, it is possible to determine which components of the permu- tation matrix P must be zero. Using this information, the optimal solution of Equation (2) can be determined.
Figure shows the experimental results. In this experiment, model merging was performed on ResNet-20 and VGG-11
3

---

Linear Mode Connectivity between Multiple Models modulo Permutation Symmetries
VGG-11
RosNet-20
Algorithm 1 STE-MM
100
100
Telt accurncy
Test aucurney
Require: Models 01. 02................. .,. ,,, the number of epochs ne
30
%
Ensure: Permutations TT....: #n-1 1: 0, < 0n for all i E [n 1n-11
8
Initialize dummy
WM
parameters 0,.
#models
#models
2: for e = 1==1... No do
(a) Test accuracics of merged models.
3:
T/ + WM 0.) for all i E [n (d:) t -UnifoxmDist((..
VGG-11
ResNet-20
4:
1))
5:
A + \/ M, A, for all i [n]
Test loss
0.75
lots
loss - L <(Σ := ( +,,0))
0.50
STF
7:
for 1 ... n 1 do
8:
grad; - Get Get (los,,, 0. + Updateee grad,)
0.00
0.00
#models
4models
10: end for
(b) Test losses of merged models.
11: end for
12: कं <- WM(0, 0:) for all i E [m -1] 1]
Figure 1: Test accuracy and loss of the merged model n-I Di (O:) when the number of models used for model merging is increased. The figures show the mean and standard deviation from three trials of model merging.
of straight-through estimators (STEs) in other discrete op- timization problems (Bengio al., 2013; Ainsworth et al., 2023), we propose a straight-through estimator for multiple models (STE-MM), method to learn permutations that can transfer multiple models into the same convex loss basin in a differentiable framework.
models trained on the CIFARIO dataset. Details on the dataset used, the training procedures for the models, and the experimental conditions for model merging, are described in Appendix B. Similar to prior studies (Ainsworth et al., 2023; Yamada al., 2024), the model width is increased to establish LMC between two models (specifically, by 16 times for ResNet-20 and times for VGG-11).
Let 01, .......................... Gn be model parameters obtained by SGD, Our goal is to find permutations Wi...., ", mn-1 such that their convex combination minimizes the expected loss:
Figure 1 shows that both STE and WM result in mono- tonic degradation of the test accuracy and loss of the merged model as the number of models increases. Notably, STE ex- hibits a sharp decline in performance. These results indicate that the conventional methods using WM and STE cannot transfer all models into the same convex loss basin.
arg min Ex L Th Nn.
AUT (0:) + -mOn
(5)
where My...y An are sampled from a uniform distribution over the simplex (i.c.. positive and summing to 1). We optimize only n 1 permutations, since the last one can be fixed without loss of generality thanks to Theorem D.
4. Permutation Search for Multiple Models
Direct optimization of Equation (5) is infeasible because permutations are discrete and non-differentiable. To ad- dress this, we adopt a STE approach: instead of optimiz- ing permutations directly, . 0,- continuous proxies. During the forward pass, we compute the best permutation गं aligning (0,) to 0 using the WM algorithm from (Ainsworth et al., 2023). In the backward pass, we compute gradients with respect to (0,) and update 6; via standard gradient descent.
In this section, we propose STE for multiple models (STE- MM) that handle multiple models simultaneously to transfer multiple SGD solutions into a single convex loss basin. First, in Section 4.1, we explain the algorithm of STE-MM. STE- MM is derived by extending STE multiple models and, therefore, uses SGD to find permutations. However, this training process becomes computationally expensive since solving the linear assignment problem (LAP) is required at each iteration of SGD. To address this issue, in Section 4.2, we propose a method to accelerate the LAP solving process.
introduce dummy parameters
This procedure is formalized in Algorithm 1. At initializa- tion (line 1), we set all Ó O,, as suggested by prior work, since improves the quality of merged models. Then, for each training epoch:
4.1. Straight-through Estimator for Multiple Models
Merging multiple independently trained models is challeng- ing because it requires solving a combinatorial optimiza- tion problem over permutations. Inspired by the success
(Line 3) For each find the permutation gद minimizing
||0 - (0.))) using the WM algorithm.

---

Linear Mode Connectivity between Multiple Models modulo Permutation Symmetries
(Lines 4-5) Sample mixture weights A1y...; An from uniform simplex.
Algorithm 2 WM
Require: Two SGD solutions 0.- = || (vec (vec(W? (')))
and
(Line 6) Compute the loss of the weighted sum of permuted models.
O,- [ (vee (W (0))
Ensure: Permutation I
1: <- (In, \/=
(Lines 7-10) Compute gradients w.r.t. x((0:) and up- date 0, accordingly.
2: while 110, (0,))l has not converged do L-13
3: LayerList +
4: while LayerList is not empty do
After training, the final permutations my are obtained by aligning the updated 0, with 02 again via WM (line 12).
5:
t P-PooRandomly(ann yLayerrLss))
6:
C
- - (((a) W/
(M)
The key insight behind this approach is that if x((0,) be- comes sufficiently close to 0, during training (i.e., (0.) x 0.). then d can be regarded as a good approximation to (O.). This idea is supported by empirical results (Sec- tion 5), showing that our method consistently transfers mod- els into a shared low-loss region, suggesting an approximate convexity in the parameter space under learned permuta- tions.
(a) e+1
P
(b)
E+1
7:
Pj f SolveLAP(C)
end while
end while
Basic idea. The idea to speed up the LAP solver is to use the optimal solution of the dual problem obtained in the previous iteration to adjust the cost matrix. Below, we provide an intuitive explanation of this idea. In the while loop on lines 2-9 of Algorithm 2, let C', P', v', and u' be the cost matrix, optimal permutation matrix, and dual variables for the f-th layer in given iteration, respectively. Similarly, let C' and P be the cost matrix and optimal per- mutation matrix in the next iteration. For simplicity, we omit the subscript & in the variables. If the iterations have sufficiently progressed, the permutation in Algorithm 2 is ex- pected to converge. Thus, we can assume that C' R C' and p' ~ P. Complementary slackness condition yields that 0 (C,, of u/,) = Pi,((Caj o0t - ut)) Based on this, we define new cost matrix C new) Ci. oi u'j
4.2. Accelerating WM
This subsection proposes a method to accelerate WM to speed up STE-MM. In particular, since STE-MM requires performing WM in every iteration of each epoch, this accel- eration leads to a significant improvement efficiency. To explain the proposed method, we first describe the original WM algorithm. Next, present the fundamental idea for solving the LAP (linear assignment problem) more quickly. Then, we describe the WM algorithm incorporating the pro- posed method. Finally, we show the experimental results of our proposed method to verify its effectiveness.
and solve the LAP for ((nww) instead of C. Note that the optimal solution (permutation) does not change when tran- sitioning from C to ((new)) Let v and u be the optimal solutions to the dual problem for C(acw), From Equation (4), Prs (C U; uy) P. of - 0;--,) u,)) = 0 must 5.3 However, if (C, o/ -(/))) e 0 holds, then v, and uy are expected to be close to zero, even with- out explicitly solving for them. Therefore, solving the dual problem for ((new) is likely to be casier than for C.
Algorithm of weight matching. For two given models O2 and 0g. the goal of WM is to find a permutation a that minimizes | π (l.) Ainsworth et al. (2023) pro- posed Algorithm 2 a method approximately solve Equation (1). First, in line 1, the permutation matrices are initialized as identity matrices (TrY) where ne rep- resents the size of the permutation matrix for é-th layer. Next, the permutation matrices are updated in lines 3-6 until the L distance| 0, - π [O,)]] between the two models converges. In line 3, a list of permutation matrices to be updated is prepared. Then, in line 5, a layer is randomly selected from this list, and it is removed from the list. In line 6, the cost matrix C' corresponding to the selected layer l is computed, and in line 7, the permutation matrix Pa that minimizes this cost is obtained by solving LAP. The ob- tained permutation matrix P. minimizes the following term: (b)
Proposed method. The accelerated WM algorithm is shown in Algorithm 3. Since most of Algorithm is the same as Algorithm 2, we will only explain the differences between them. The first difference is that Algorithm takes the dual variables (ve) and (ue) as inputs and also returns the dual variables obtained at the end of the algorithm. This is used to accelerate STE. Specifically, in Algorithm 1, WM function is replaced with the proposed accelerated WM al- gorithm, and the dual variables of the i-th model obtained from the previous epoch (or the previous iteration of SGD for finding permutation) are passed to the replaced WM func-
e+1 P.T 112 + []W((a) - (. (b) PT,112 The most time-consuming step in WM is solving the LAP.
[ Pe+1 !!t))
8+1
For instance, the computational cost of solving LAP with the Hungarian algorithm is O(n3). Therefore, we propose a method to solve LAP more efficiently.
5

---

Linear Mode Connectivity between Multiple Models modulo Permutation Symmetries
VGG-11
ResNett20
5. Merging Multiple Models
Ellaped Gime fsl
104
Eltased time (d]
10
Ainoworth et al.
This section conducts some experiments using multiple SGD solutions to confirm the effectiveness of STE-MM. Since verifying directly that all models belong to a convex basin is difficult, we investigate the test losses of merged models within scope that can be easily verified. In the next section, a deeper analysis is conducted, focusing on the sharpness of the loss function around the merged models.
101
10
10
10
STE
WM
STE:
WM
Figure 2: Processing time of WM and STE.
Algorithm 3 Accelerated WM
Section 5.1 first examines the losses of merged models ob- tained by applying permutations. Then, in Section 5.2, we examine the losses between all pairs of models. These losses are expected to be comparable to the original models. Fi- nally, Section 5.3 experimentally demonstrates that larger model widths lead to lower losses in the merged models.
Require: Two SGD solutions 6, and O,. dual variables
(ve)e and (ue)e
Ensure: Permutation ग,, dual variables (ve)) and (ue)e
1: π +- (fn
2: while ||0, 7 T(6b)l has not converged do
3: LayerListt < 21,2,.... I L- 1}
4:
while LayerList is not empty do
5.1. Merging All Models
5:
8 P--PoRaadomyy(Layerlll
Figure 3(a) shows the test accuracy of the merged model. Specifically, given n SGD solutions 01, 02..... On and the coresponding permutations Ti, T....., Tr (where, in STE- MM. R. (In)) holds), presents the test accu- racy of the merged model 1.4 Σ, (O,). For reference, the figure also includes results using two other methods for model merging: MergeMany, proposed in Appendix A.10 of Ainsworth al. (2023), and cycle consistent multi-model merging (CCMM), proposed by Crisostomi et al. (2024). Both methods aim to find permutations Tı, TT2..... ... nr that minimize the sum of T,2 distances between all model pairs, Eis || (0.) Ty (1,) Additionally, Figure 3(b) shows how much the test loss of the merged model increases com- pared to the original models' losses.
6:
C
-W( (0)
P 1
(a)
(0)
111
7:
(Ct,) t (C. - ue,1 -.,)
8:
Pe. u oy t -SolveLAPWithDuals..
9:
Up + 0y + o/; us + us + u/
10: end while
11: end while
tion, enabling faster execution. In fact, our implementation of STE-MM adopts this approach. Note that, when the first iteration of STE-MM is executed, ue and Uy input to the WM are set to 0. The second difference is in line 7, where the dual variables are subtracted from the cost matrix to get ((new), This step has already been explained above. The third difference is in line 8, where not only the permuta- tion matrix but also the dual variables un and v obtained by solving the LAP are returned. Then, in line 9, the dual variables V2 and us are updated by adding these values. By using this adjustment, the dual variables Uy and obtained in line 9 match the optimal solutions of the dual problem for the original cost matrix C.
First, Figure 3(a) shows that the test accuracy of the merged model increases with the number of models with STE-MM, whereas it decreases with other methods. This difference is likely due to the fact that the objective function of STE-MM directly minimizes the loss values, whereas the objective functions of MergeMany and CCMM focus on reducing the distances between models. Next, Figure 3(b) shows that, ex- cept for the MLP models trained on MNIST, increasing the number of models leads to increase in the test loss. No- tably, the increase is more pronounced in MergeMany and CCMM. While the loss of the merged model also tends to in- crease with the number of models in STE-MM, the increase in test loss value becomes small as the number of models grows. In fact, as shown in Figure 3(a), the test accuracy improves as the number of models increases. suggesting that the increase in test loss may converge. Furthermore, the difference between the original models' losses and the merged model's loss is close to zero in STE-MM, indicating that the merged model's 's loss is sufficiently small.
Experimental comparison. Figure 2 shows the process- ing time of WM and STE to find permutations for two models, with and without the proposed acceleration of the LAP solver described in Section 4.2. The figure presents the mean and standard deviation of processing time when a permutation search is conducted three times. For STE, the number of epochs was set to one. From the figure, we can see that both WM and STE with our acceleration method are at least approximately ten times faster than those without it, regardless of the model architecture. In particular, when finding permutations for ResNet-20 models, our method reduces the processing time to 1/40 of the original.
6

---

Linear Mode Connectivity between Multiple Models modulo Permutation Symmetries
MLP MNIST
MLP FMNIST
VGG-11 CIFARIO
ResNet-20 CIFARIO
100
100
100
100
Tox RONiRAyyy
Tos a0cuECy
Toe sOUEERY
90
90
90
STE-MM MargsMaary
. CUMM
Text
Pec
80
30
80
10
2
Amedelss
Amoddls
#mudek
(a) Test accuracies of merged models.
MNIST
MLP FMNIST
VGG- CIEARIO
ResNet- CITARIO
Loss dif.
0.00
Loss dit.
0.00
0.00
0.00
-0.25
STEAMM
-+-
DCMM
-0.25
-0.25
-0.25
MageMaary
0.50
10 0.50
10
10
4
10
Anndlels
Amasllls
Ansakkl
Anssch
(b) Difference between the test loss of merged model and the average test loss of original models: L(2 E, (0,)) - E. ((O)).
Figure 3: Experimental results of merged models π Eie(n) Ty (0,) when the number of models increased. The figures
show the mean and standard deviation from three trials of model merging.
MLP MNIST
MLP IMNNST
VGG- 11 CIFARIO
ResNet 20 CITARIO
0.25
0:25
0.25
0.25
Low.dit.
0.00
Laxeder.
0.00
Loo
0.25
0.25
-.255
-0255
MaceeMazy
-0.50
-0.50
-0.50
-0.90
Amsdcls
Amadcls
Ansokk
Figure 4: Difference average test losses between the merged models (for all model pairs) and the original models: n(n-1) Σies [()(T, + T, ((0,).
5.2. Merging Each Model Pair
VGG-11 CIPARIO
Re<Net-20 CIFARIO
MM
CUMM
Next, we investigate the loss when merging each model pair. Figure 4 illustrates the difference between the aver- age loss of the merged models for all model pairs and the loss of the original models (i.e., n[n-1] Eiey (T, E,E,,O (().)). For MLP models trained FM-
MargeMany
T, (0,0 - E, NIST, it was observed that the average loss of the model pairs increases when using STE-MM. However, the differ- ence in loss remains below O, and the amount of the increase in loss value diminishes as the number of models grows. This suggests that increasing the number of models may not pose issue when transferring multiple models to the same low-loss basin. For MLP, VGG-1 1, and ResNet-20 models trained on MNIST or CIFAR-10, the impact of increasing the number of models on the loss is very small, with the difference in loss remaining very close to O. Consequently, this indicates that the loss barrier between any model pair is not particularly significant.
Wilth mullilier
Wilt multiplicr
Figure 5: Difference between the test loss of the merged model and the average test loss of the original models when varying model width.
merged model of all the original models and the average loss of the original models when ten SGD solutions are merged. The figure shows that, for all methods, the wider the model, the closer the loss difference approaches zero. This result is the same as the trend observed in previous studies on merging two models, indicating that increasing the model width also facilitates LMC in multiple models.
5.3. Model Width
6. Sharpness of Merged Model
Finally, we examine the impact of model width on the test loss of the merged model. Figure 5 shows the effect of model width the difference between the loss of the
This section investigates the sharpness of the loss for the merged model. If the permuted models belong to the same low-loss convex basin, the sharpness of the merged model's
7

---

Lincar Mode Connectivity between Multiple Models modulo Permutation Symmetries
MLP MNIST
MLP FMNIST
VGG-11 CIFARIO
RosNet-20 CIFARIO
20
STE-MM
5.0
2.5
10
0.0
0.5
1,0
1.5
2.0
0.0
0.5
1,0
1.5
2.0
0.0
0,5
1.0
1.5
2,0
0.0
0.5
1.0
1.5
2.0
Disttnee
Distsaao
Figure 6: Histogram of L distances of all model pairs using STE.
MLP MNIST
MLP FMNIST
VGG-11 CIFARIO
ResNet- CIFARIO
100
STE MM
50
40
00
80
100
120
0
20
40
60
30
100
120
20
40
60
80
100
120
20
40
60
800
100
120
Degree
Degroe [*]
Degree (1]
Degree (*]
Figure 7: Histogram of degrees of all model pairs using STE.
loss can be expected to be smaller than that of the original models. First, in Section 6.1, we examine the geometric rela- tion of models in the parameter space after applying the per- mutations and reveal that the models form an approximately regular simplex. Next, in Section 6.2, we measure sharpness using the Hessian matrix of the models and demonstrate that the sharpness of the merged model is smaller than that of the original models when using STE-MM.
n. dimensions, the condition for these models to form a con- vex low-loss basin is that the losses at all points within the regular simplex are small. However, as the dimensionality increases, the space of points where loss needs to be verified grows exponentially. making it impractical to examine the loss at all points. Therefore, in this subsection, tigate the sharpness of the loss at the center point, which corresponds the merged model. If multiple models form convex low-loss basin, it is expected that the sharpness of the loss at the center will be smaller than the sharpness at the vertices (i.e., the trained models).
inves-
6.1. Geometric Relation of Models
Figure 6 shows the L distances between each pair of n per- muted models x (01), T2 (22). ... Tr (0x) (i.e., (0.) T, (I)) This figure illustrates the results for = 10 mod- els when STE-MM is used to find pernutations. The experi- mental results for other permutation methods are shown in Figure 9. Since there are n(n-1) 45 model pairs, the 2 distances are presented as a histogram.
Definition of sharpness. Although there are various defi- nitions of sharpness, we focus on the trace of the Hessian matrix H of the loss function, tr H, which measures the sharpness of the loss in random direction (Wen et al., 2023). tr H has good theoretical properties. For example, under certain assumptions, sharpness- aware minimization (Foret et al., 2021) and label noise SGD induce a small trace of the Hessian (Bartlett et al., 2023; Damian et al., 2021; Wen et al., 2023). However, using tr H directly presents an is- sue for rectifier neural networks (i.e., those using ReLU as activation function), as its value is not unique (Li et al., 2018; Kwon et al., 2021). Specifically, tr H is not invariant to weight rescaling, even though such rescaling does not change the input-output functionality. To address this issue, instead of H, we use tr(H O 00T), which known to asymptotically match the average-case sharpness (An- driushchenko et al., 2023). Experimentallyy, to compute tr [ @ 0000) ) for model 0, we use the following equation:
2
The horizontal axis in Figure 9 represents the ratio of the 2 distance for each model pair to the av- erage T distance across all pairs (i.e., (0,) T m,(0))l// m1 Eie (0,)) T, (l) Figure 9 shows that the distances for all model pairs are close to the average distance. This tendency is particularly strong for VGG-11 and ResNet-20, where the distance ratios are very close to 1. This result suggests that the permuted models are located at approximately equal distances from each other, forming an n-dimensional regular simplex. Figures 7 and 10 also shows the angles between all possible triplets of models, which supports the conclusion that the models form a regular simplex.
tr H C T = Ez(z diag( (0) H diag (0)).
(6)
where z is a Rademacher random vector, and diag(O) denotes a diagonal matrix with the elements of 0 as its diagonal entries. The proof of this equation is pro- vided in Appendix D.2. The product of any vector
6.2. Sharpness
Since the models x1 (01), 72(2),..... F. (0x), after apply- ing permutations, form approximately regular simplex in
8

---

Linear Mode Connectivity between Multiple Models modulo Permutation Symmetries
MLP MNIST
MLP FMNIST
VGG-11 CIFARIO
ResNet-20 CIFARIO
0.6
STE SM
-+-- CCMM Bastinn
Shurpess
04
Mingen MrgeMMry
Shappes
Sharpees
Sharpess
0.2
10
0.0
2
10
Amadels
Arnodels
Figure 8: Sharpness of merged models.
1 and the Hessian matrix H (i.e., jHH) can be com- puted using torch. autograd, functional vhp in PyTorch'. This enables the evaluation of tr(H @ 00
models and simple datasets. An important direction for fu- ture work is to investigate the applicability and effectiveness of the proposed method in more practical settings involving larger models and more complex data. Moreover, although STE-MM successfully discovers suitable permutations for merging multiple models, the reason for the existence of such permutations remains unelear. Clarifying this from the perspective of the learning dynamies of SGD would con- tribute to deeper theoretical understanding. In addition, it is widely observed that satisfying LMC using permuta- tion symmetries of NNs often requires increasing the model width. Exploring whether LMC can be achieved without ex- panding the width, potentially by developing more advanced permutation strategies or leveraging inherent architectural properties, remains an important open question.
Experimental results. Figure 8 shows the sharpness of the merged models. In the figure, "Baseline" represents the average sharpness of the models used for merging. For MLP models trained on MNIST and FMNIST, Figure 8 shows that the sharpness decreases the number of models increases, regardless of the methods. Notably, the sharpness of the merged models is smaller than that of the original models. On the other hand, for VGG-11 and ResNet-20 models, as the number of models increases, the sharpness monotonically increases for MergeMany and CCMM, while decreases for STE-MM. This indicates that when models transferred using the permutations found by STE-MM, the loss in the center of those models is, on average, flat. Thus, this indicates that STE-MM would successfully gather the models into an approximately convex low-loss basin.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
7. Conclusion
In this paper, analyzed whether multiple models can be gathered into a single approximately convex basin by using the permutation symmetries of neural networks. First, we experimentally showed that the performance of merged models decreases when the number of models is increased in the conventional permutation search method for two mod- els. Next, we proposed a permutation search method using a straight-through estimator for multiple models (STE-MM). We then demonstrated the effectiveness of STE-MM using MLP, VGG-11. 11, and ResNet-20 models trained on MNIST, FMNIST, and CIFAR10. Finally, we investigated the sharp- ness of the loss function and found that the sharpness of the merged model monotonically decreases when using STE- MM, which indicates that the multiple models are gathered into a single approximately convex basin.
References
Ainsworth, S., Hayase, J., and Srinivasa, S. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Represen- tations, 2023. URL https: / openreview .nt// Torum?id=CQsmMYymPPP
Andriushchenko, M., Croce, F., Müller, M., Hein, M., and Flammarion, N. A modern look at the relationship be- tween sharpness and generalization. In Krause, A., Brun- skill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Confer- ence on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 840-902, PMLR, 23- .mlr.
29 Jul 2023. URL https: press / w21///22222/// 12202/andrrii http///ddriuuscchenn22.a... htmll...............m..mmmm.....
These findings provide promising foundation, but several important questions remain open for future work. This study focused on image classification tasks using relatively small
Bartlett, P. L., Long, P. M., and Bousquet, O. The dynam- ics of sharpness- aware minimization: Bouncing across ravines and drifting towards wide minima, 2023. URL https: wtwp/aiivvco//c/ www.aawwwwwww...../r//ea.e.eaeee
htt.://rwcc...rr.o///s//taaaaeae generated//torch.auutee aut.orrr........o............... fun.tiiiooonooiiiiiiiiii html
functionall
www./oo/////aaaeoooo 22100. 01513.
9

---

Linear Mode Connectivity between Multiple Models modulo Permutation Symmetries
Bengio, Y., Léonard, N., and Courville, A. C. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, 13011338833323333 2013. URL hattp://arxiv/ ottp//ivvo.ggoaoooooooo abs 1308. 3432.
ceedings of the 37th International Conference on Ma- chine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 3259-3269. PMLR, 13-18 Jul 2020. httphtttss//ppcceeeeeeeee v1 1119/franke httmlllll..a...........mm
mlr press/
Crisostomi, D., Fumero, M., Baieri, D., Bernard, F., and Rodolà, E. M Cycle-consistent multi-model merg- ing. arXiv preprint arXiv:2405.
Freeman, C. D. and Bruna, J. Topology and geom- etry of half-rectified network optimization. In In- ternational Conference on Learning Representations, 2017. URL https: /tsp::///ppnevview net/ formm? i id-BkCFWVegx.
2024.
Damian, A., Ma, T., and Lee, J. D. Label noise sgd provably prefers flat global minimizers. In Ranzato, M., Beygelz- imer, A., Dauphin, Y., Liang. P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 27449-27461. Curran Associates, Inc., 2021. URL https: https::/prrcced..gg....em neurip... cc cc/pappee paper_fi [eape__flleespppee//2/////1/ FFFf440c28c1190eeee1111 pdf.
Garipov, T., Izmailov, P., Podoprikhin, D., Vetrov, D. P., and Wilson, A. G. Loss surfaces, mode connectivity, and fast ensembling of dnns. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Gareett, R. (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL htpps///poocc...gg neurips. cc/paper cc/papee__fileepppper////11111B- Paper.
Paper.
Draxler, F., Veschgini, K., Salmhofer, M., and Hamprecht, F. Essentially no barriers in neural network energy landscape. In Dy, J. and Krause, A. (eds.), Proceed- ings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learn- ing Research, pp. 1309-1318. PMLR, 10-15 Jul 2018. BRLhttps:: pttppppooccnnc.ccpee mlr wwwppprssss////// draxler188 8a. .8mmmml.
pdf.
Hutchinson, M. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Statistics Simulation and Com- putation, 19(2):33-455, 1990. doi: 10.1080/ 036109190088 12866. URL https: //o::///oiii org/10. 1080 0080003360999 0000000100000080660000
Entezari, R., Sedghi, H., Saukh, O., and Neyshabur, The role of permutation invariance in linear mode connectivity of neural networks, 2022.
Ito, A., Yamada, M., and Kumagai, A. Analysis of lin- ear mode connectivity via permutation-basee weight matching: With insights into other permutation search methods. In The Thirleenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https: / //oenrevieww net www.mmiidweereee.....er....
Ferbach, D., Goujaud, B., Gidel, G., and Dieuleveut, A. Proving linear mode connectivity of neural net- works via optimal transport. In Dasgupta, S., Mandt, S., and Li. Y. (eds.), Proceedings of The 27th Interna- tional Conference on Artificial Intelligence and Statis- tics, volume 238 of Proceedings of Machine Learn- ing Research, pp. 3853-3861. PMLR, 02-04 May 2024. httsttps:///// / p/poceeedgs mlr. .plesp/ v238/ wtt.//ebbbahaaaaaaaaaea html.
Jonker, R. and Volgenant, T. A shortest augmenting path al- gorithm for dense and sparse linear assignment problems. Computing, 38:325-340, 1987. URL htt /api. semanticschole org/ Corpus 7806079.
Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations, 2021. URL h //tp///veeeeeeeeee//eeee/e./ net/ forum? (etp:ummiid--11 www.d:6mmmmmmpppp.mmm
Jordan, K., Sedghi, H., Saukh, O., Entezari, R., and Neyshabur, B. REPAIR: REnormalizing permuted ac- tivations for interpolation repair. In The Eleventh In- ternational Conference on Learning Representations, for//orum?
2023, URL httpp::///ppeneeeee i id=gU5sJ62ggcX.
Frankle, J. and Carbin, M. The lottery ticket hypothe- sis: Finding sparse, trainable neural networks. In In- ternational Conference on Learning Representations, forum?
Konečný. J., McMahan, H. B., Ramage, D., and Richtárik, P. Federated optimization: Distributed machine learning for on-device intelligence, 2016.
2019. URL attpp:://penreviie id=rJ1-b3ReF7
Koopmans, T. C. and Beckmann, M. Assignment problems and the location of economic activities. Econometrica, 25(1):33,76, 1957. ISSN 00129682, 14680262. URL http:://ww.jjstor... jstor. .aweerorgg/eeee wttpgg/stallee/ee 10907//1//9997722
Frankle, J., Dziugaite, G. K., Roy, D., and Carbin, M. Linear mode connectivity and the lottery ticket hy- pothesis. III, H. D. and Singh, A. (eds.), Pro-
10

---

Lincar Mode Connectivity between Multiple Models modulo Permutation Symmetries
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
learning- In Wallach, H., Larochelle, H., Beygelz- imer, A., d'Alché-Buc, E., Fox, E., and Gamett, R. (eds.), Advances in Neural Information Process- ing Systems, volume 32. Curran Associates, Inc., 2019. URL httpps///proocedngg...errr cc cc/pppee__iieee/pppr///////LLL 0 Paper.. pdf.
Kuditipudi, R. Wang. X., Lee, H., Zhang, Y., Li, Z.,, Hu, W., Ge, R., and Arora, S. Explaining landscape connectivity of low-cost solutions for multilayer nets. In Wallach, H., Larochelle, H., Beygelzimer, A., d' d'Alcéé-Bue, F., Fox, E.,, and Garnett, R. (eds.). Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL httpss//prcccdddngg.s neereeeeeeddddd....ee.......... cc
Paper.
Nguyen, Q. On connected sublevel sets in deep learning. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceed- ings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learn- ing Research, pp. 4790-4799, PMLR, 09-15 Jun 2019. URL https: /ttp:::///rrceeeeeeeeee pttpp//ooceennnn.e/eee/p ml pwwss/v97. nguyen1 9a. html.
1acd376ff550000000005557ooo6e333 pdf.
Paper.
Kuhn, H. W. The hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(1-2):83- 97, 1955. doi: http::/dbb.oogg//000000m0ass0000 URL https://onlinell// doi/abs /0t0 htt.://///.....000000000000
wiley. com/
Nguyen, Q., Mukkamala, M. C., and Hein, M. On the loss landscape of class of deep neural networks with no bad local valleys. In International Conference on Learning Representations, 2019. URL https:: /opp//veeeeeeeeeeeeeeeeeeee net / Hug
Kwon, Kim, J., Park, H., and Choi, I. K. Asam: Adaptive sharpness-aware minimization for scale-invariant learn- ing of deep neural networks. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Confer- ence on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 5905-5914. PMIR, 18-24 Jul 2021. http:: htttpss:pppocceeinggss mll.. press/vee v1 39/ kwon21b.
Peña, F. A. G., Medeiros, H. R., Dubail, T., Aminbeidokhti, M., Granger, E., and Pedersoli, M. Re-basin via implicit sinkhorn differentiation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 20237-20246, June 2023.
Lecun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient- based learning applied document recognition. Pro- ceedings of the IEFE, 10.1109/5.726791.
Sahni, S. and Gonzalez, T. P-complete approximation problems. J. ACM, 23(3):555 565, jul 1976. ISSN 0004-5411. doi: 10. 10.1145/321958..21997 URL https: / //dbi. 1141 3215553215558 32111199....2........
1998. doi:
Li, H., Xu, Z. Taylor, G., Studer, C. and Gold- stein, T. Visualizing the loss landscape of neural nets. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnetr, R. (eds.), Advances in Neural Information Process- ing Systems, volume 31. Curran Associates, Inc., 2018. URL htttps:/pproceednngg neurips. cc hccpppppr_ffieeeepppee///////// (llbbbbbeebb00006666677776616613 Papee. pdf.
Singh, S. P. and Jaggi. M. Model fusion via optimal transport. In Larochelle, H., Ranzato, M., Had- sell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, vol- ume 33, pp. 22045-22055. Curran Associates, Inc., 2020. URL https://prroeeddnng...eee cc ccc/ppppr__ffie/ppppe////////// 1b257769F54444046cce223333000333 pdf.
van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K. Wavenet: A generative model for raw audio, 2016.
McMahan. B., Moore, E., Ramage, D., Hampson, S., and Arcas, B. A. y. Communication-Efficiett Learn- ing of Deep Networks from Decentralized Data. In Singh, A. and Zhu, J. (eds.), Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Proceedings of Machine Learn- ing Research, pp. 1273-1282. PMLR, 20-22 Apr 2017. URL UtLsttps:// http://///occeeeeeeeeeee mlr press/v54/ memahan17e html.
Vaswani, A., Shazeer, N.. Parmar, N., Uszkoreit. J., Jones, L.. Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https::/proceediini wwwphinngn..uuurreereee httpppperfllee///eeppppppp ftappr__filee//pper///1/////
Nagarajan. V. and Kolter, J. Z. Uniform convergence may be unable to explain generalization in deep

---

Lincar Mode Connectivity between Multiple Models modulo Permutation Symmetries
(l0l pdf.
Paper.
Venturi, L., Bandeira, A. S., and Bruna, J. Spurious valleys in one-hidden-layer neural network optimization land- scapes. Journal of Machine Learning Research, 20(133); 1-34, 2019. URL v20/18- ..
hwt.m...orpppsss.i.////mm papers/
Wang, H., Yurochkin, M., Sun, Y., Papailiopoulos, D., and Khazaeni, Federated learning with matched averaging. In International Conference on Learning Reprexentations, 2020. URL https: http//pppeeeeeeeeeeeeeeeee forum? id=Bklug1SFDS.
Wen, K., Li, Z., and Ma, T. Sharpness minimization algo- rithms do not only minimize sharpness to achieve better generalization. In seirnt- Conference Neural Information Processing Systems, 2023. URL. https: //openrevien www.e.eeeeeeee.e..e.........e..e
Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-opees, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., and Schmidt, L. Model soups: averaging weights of multiple fine- tuned models improves accuracy without increasing in- ference time. Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Pro- ceedings of the 39th International Conference on Ma- chine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 23965-23998 PMLR, 17-23 Jul 2022. URL http://///ooeogeee/eeeeee www..m..........mm...........w.. press/ v1 62 htt///oortaann2222mmm html.
Xiao, H., Rasul, K., and Vollgraf, R. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms, 2017.
Yamada, M., Yamashita, T., Yamaguchi, S., and Chijiwa, D. Toward data efficient model merging between dif- ferent datasets without performance degradation. In The 16th Asian Conference on Machine Learning (Conference Track), 2024. URL forum? 1=w-eddeeeeeeae... aeSAn..
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., Du, Y., Yang, C.,
Chen, Y., Chen, Z., Jiang, J., Ren, R., Li, Y., Tang, X.,
Liu, 2., Liu, P., Nie, J.-Y., and Wen, J.-R. survey of large language models, 2023.
Zhou, Z., Yang, Y., Yang, X., Yan, J., and Hu, W. Going be- yond linear mode connectivity: The layerwise linear fea- ture connectivity. Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL ht tttps: /openreview www.:/ee//eeeeeeee wwwp/eeuu//eee/eeeee 2et://oommm2==00000 ii=
12

---

Lincar Mode Connectivity between Multiple Models modulo Permutation Symmetries
A. Additional Related Work
(Linear) Mode Connectivity. Several studies (Garipov et al., 2018; Draxler et al., 2018; Freeman & Bruna, 2017) have demonstrated that different neural networks (NNs) can be connected through simple nonlinear paths with negligible loss increase. Nagarajan & Kolter (2019) were the first to discover that, in the case of models trained on MNIST with the same random initializations, solutions obtained via stochastic gradient descent (SGD) can also be connected by linear paths while maintaining an almost constant loss value. Subsequently, Frankle et al. (2020) empirically showed that linear mode connectivity (LMC) does not always hold between two SGD solutions, even when they share the same initialization, as it depends on the dataset and model architecture. However, they also demonstrated that if a model is initially trained for certain duration and then used as the starting point for training two independent models, these models are linearly mode-connected. Furthermore, Frankle et al. (2020) explored the relationship between LMC and the lottery-ticket hypothesis (Frankle & Carbin, 2019). Entezari et al. (2022) conjectured that LMC holds, with high probability, when considering permutation symmetries in hidden layers. Later, Ainsworth et al. (2023) proposed neuron alignment as bipartite graph matching problem and solving it approximately. Following this, Peña et al. (2023) introduced the use of Sinkhorn's 's algorithm to directly solve the WM problem. Although several works (Venturi et al., 2019; Nguyen et al., 2019; Nguyen, 2019; Kuditipudi et al., 2019) have investigated nonlinear mode connectivity, theoretical analyses of LMC remain limited. Ferbach et al. (2024) provided an upper bound on the minimal width of the hidden layer to satisfy LMC if the independence of all neurons' weight vectors inside a given layer holds. Zhou et al. (2023) introduced the concept of layerwise linear feature connectivity (LLFC) and established that LLFC implies LMC. Ito al. (2025) demonstrated that the top singular vectors of the parameters obtained by SGD play an important role in satisfying LMC, especially when permutations are found using WM. There are few studies merging multiple models using permutations. Ainsworth et al. (2023) proposed MergeMany, which extends WM to multiple models. Crisostomi et al. (2024) proposed a method for searching permutations by minimizing the sum of distances between all model pairs as the objective function using the Frank-Wolf algorithm. However, as shown in Section 5, these methods have the problem that the performance of the merged model worsens as the number of models increases.
matthhin (WM) method by formulating
Model Merging, Model merging and federated learning are relevant topics in the study of LMC. McMahan et al. (2017) and Konečný et al. (2016) are the first to introduce federated learning, a technique where models are trained on partitioned datasets. Wang et al. (2020) proposed a federated learning approach that involves permuting individual model components before averaging their weights. Similarly, Singh & Jaggi (2020) developed a model merging technique that aligns model weights using optimal transport, a method conceptually similar that of Ainsworth et al. (2023). While Singh & Jaggi (2020)'s approach is designed for model fusion and performs worse than Ainsworth al.'s method, it can still be categorized as an LMC-based technique due to its use of hard alignments within the same architecture. Wortsman et al. (2022) introduced a eeight-averaging strategy that enhances test accuracy without increasing inference costs, distinguishing it from traditional ensemble methods.
B. Experimental Settings
This section describes the experimental setup for training neural networks to obtain SGD solutions. In addition, we describe the hyperparameters for permutation search methods. Three datasets were used in this study: MNIST (Lecun et al., 1998), Fashion-MNIST (FMNIST) (Xiao et al., 2017), and CIFARIO 10 (Krizhevsky et al., 2009).
All experiments were conducted on a Linux workstation with two AMD EPYC 7543 32-Core processors, eight NVIDIA A30 GPUs, and 512 GB of memory. The PyTorch 2.5.13, PyTorch Lightning 2.4.0*, and torchvision 0.20.1 libraries were used for model training and evaluation.
B.1. Model Training
MLP on MNIST and FMNIST. Following the approach outlined in (Ainsworth et al., 2023), we implemented a multi- layer perceptron (MLP) consisting of three hidden layers, each with 512 units. The ReLU activation function was employed for the hidden layers. For training on the MNIST and FMNIST datasets, the Adam optimizer was utilized with learning rate of 10-3 3, The batch size was fixed 512, and training was conducted for a maximum of 100 epochs. We did not
'nttps: http::://ytt.cceeeee org/
httep:://yycrrch..r////iiio///// http////aaaaleeeeeeeeeeeeee html
13

---

Lincar Mode Connectivity between Multiple Models modulo Permutation Symmetries
MLP MNIST
MLP FMNIST
VGG-I1 CIFARIO
MeNNe-22CIIARR
20
MergeMaayy
0.0
0.5
1.0
2,0
0.0
0.5
1.5
2.0
0.0
0..5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Tiistance IEDO
DElO
Didtmorr
ratio
(a) MergeMany.
FMNIST
VGG-11 CIFARID
CCMN
5.0
0.0
0.5
1.0
1.5
2,0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
(b) CCMM.
Figure 9: Histogram of L distances of all model pairs.
use learning scheduler.
VGG-11 and ResNet-20 on CIFAR10. We utilized the VGG-11 and ResNet20 architectures (Ainsworth et al., 2023). To accomplish linear mode connectivity (LMM)), we increased the widths of VGGI1 and ResNet20 by factors of and 16, respectively. As described in (Jordan al., 2023), we used the training dataset repair the BatchNorm layers these models during model merging. Optimization was conducted using SGD with learning rate of 0.4 and weight decay of 5 x 10-4. 4. The batch size and maximum number of epochs were set to 500 and 100, respectively. The following data augmentations were performed during training: random 32 x 32 pixel crops, and random horizontal flips. Based on the GitHub repository of (Jordan et al., 2023),, the learning scheduler was prepared according to the following code:
execute this code, you need to import the packages as shown below in advance. import numpy np
import torch
# Here's the code for generating the learning rate scheduler.
# num_epochs: the number of epochs, opt: optimizer
ir_sch = inp.nntrp arange (1 + num_epochs), [0, 5, num_ num_epochs), [0, 1, 0])
sch
1raetiimm._scchheeeerr LambdaLR (opt, lr_sch. getsitee
B.2. Permutation Search
WM and STE were implemented following the GitHub repository of (Ainsworth et al., 2023)?. . For STE and STE-MM, the learning rate, number of epochs, and batch size were set to 0,001, 10, and 256, respectively. As for CCMM, the source code from the authors' publicly available GitHub repository was used
C. Additional Experimental Results
C.1. 2 distance between all model pairs using MangeMany and CCMM
Figure 9 shows histograms of the 7,2 distance ratios between all model pairs for MergeMany and CCMM. The figure indicates that the distances between model pairs are very close to their mean values, suggesting that they are nearly equivalent.
'https: http:::///ithii http:/hh...coceeeeeeorrrraaaaaaa "htpps: ://i thub. com com/saamella/git-reeetee 'https://oe thub. http:/.com/rrrsotttooooolleeeeee
14

---

Lincar Mode Connectivity between Multiple Models modulo Permutation Symmetries
MLP MNIST
MLP FMNNST
VGG-11 CIFARIO
ResNet-20 CIFARIO
100
MergeMang
20
40
0/0
80
100
120
20
90
00
30
1000
120
20
30
60
80
100
120
20
40
60
80
100
130
Degnne
Degnne (?)
Degron
(a) MergeMany.
MLP MNIST
MLP FMNIST
VGG-I1 CIFARIO
RecNet- CIFAR10
10
20
Degres
Degroe (*]
Degree (1]
Degree [*]
(b) CCMM.
Figure 10: Histograms of angles formed by all mode triplets.
C.2. Angle of each triplet taken from the models
Figure 10 represents the histogram of the angles of all triplets from the models after applying the permutation. From the figure, most of the angles formed by the triplets are close to 60 degrees, suggesting, described in Section 6.1, that the models after applying the permutation are close to regular simplex.
D. Proofs
D.1. Finding n permutations is sufficient
This subsection proves the following theorem:
Theorem D.1. Let g Relpanee + be a function from the parameter space to the real numbers and invariant to permutations (i.e., for any parameter and permutation æ. g(T((O) = g(0) holds). Then, for any n. models O1...., On. R. there exist n permutations ,.,...,,7 T such that g(Eie(w) (0.)) 9 ((innn- (O, S,, holds. Here, Ti...., - are independent of the function g.
any permatations TI, T.....y Wn, and any scalars 81; 81:..., 8n
Note that the loss function L and 2 norm are invariant to permutations.
Proof. We first prove the linearlity of permutaion (P)))e(4). For any two parameters -1/2 (vec (We) || by) and 0' -li (vec(W) bb), and scalers 8, R, consider r(s0 s'0'). From definition, since sO s'0' I Le= ((vec(sWe+ s'W))) || (sb: s'b))) holds, have =ss0 + s'8') = 1 | (sP.W, || + 1|=_1 (PP.W.PTT &
(&Pbbe + &PPb))
(7)
(8)
=lll (P.W,P... || P.b.)) + + (P.W P.b.)) ST(( s' (8').
(9)
(10)
which indicates that the permutation π is linear function.
Then, prove that there exists permutations #1, nh.,... ... The1 such that we have
g
Σ ic(m]
( =
Σ
s,,)(0,) 0n
(11)
iE|n-1]
15

---

Lincar Mode Connectivity between Multiple Models modulo Permutation Symmetries
From the linearity of permutations and the invariance of y with respect permutations, have
Σ ():)( ))
1) = Σ
0
Σ (0.) iE(m
9
nn nn
-9 n Σ
St TT((0))
iEIml
iEin]
(12)
Thus, the permutations T/ ,, o 11, 22 n 22..... ................................ n- an Ān-1 permutations oi...., T are independent the function g.
satisfy Equation (11). addition, the
D.2. Proof of Equation (6)
We prove the following theorem.
Theorem D.2. Let z be a Rademacher random vector, 0 be parameters of an NN model, and H be the Hessian matrix of the loss function at the model Then, we have
tr H @ 00 = Ex (z diag(O) ()) H H(z diag(0))
(13)
Proof. With the Huchinson trace estimator (Hutchinson, 1990), we have
tr H @ (00) TT) )) ir(H diag( 0) diag 0)) = ir(diag(0) H diag(())
(14)
Eg=² diag(O) H diag(0) (0) (diag(())) H diag ()).
(15)
16


Reviews:
[{"review_id": "47wZY53Rgq", "reviewer": "Reviewer_AEKC", "summary": "This paper focuses on the linear mode connectivity between neural networks (NNs) trained using stochastic gradient descent (SGD). First, it shows that existing permutation search methods perform poorly when more than two models are involved. To address this issue, the authors propose a novel search method, the Straight-Through Estimator for Multiple Models (STE-MM). Empirical results show that this method is both effective and efficient in improving linear mode connectivity among multiple models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reading our paper carefully and for your constructive comments.\n\n> However, other merging methods, such as TIES merging and DARE, are also widely used and have been shown to be effective. Additionally, due to computational cost considerations, simple averaging remains a common approach in multiple-model merging.\n\nThank you for your comment. While those methods are indeed important prior works, their assumptions differ significantly from ours. Specifically, TIES and DARE propose methods for merging multiple fine-tuned models based on a shared pre-trained model. In contrast, we focus on merging models that have been trained from scratch using different random seeds. Previous studies, such as model soups (Wortsman et al. 2022), have shown that simply averaging the weights of fine-tuned models originating from a common base model can improve performance. However, when merging models trained from scratch with different seeds, it has been observed that naive weight averaging can substantially degrade performance. Our method addresses the more challenging scenario of merging such models effectively, which we consider one of our key contributions.\n\n> The clarity of Figure 5 could be improved. \n\nWe apologize for Figure 5's poor visibility. As you pointed out, we will revise it in the camera-ready version to improve its readability.\n\n> Typos: 059 (right column) model; 061 (right column) denotes.\n\nThank you for pointing out the typos. We will thoroughly proofread the entire manuscript to ensure that all typos are corrected.\n\n> Could STE-MM also assist in determining the optimal hyperparameters for different models? For instance, in model merging, where different models have distinct weights, can STE-MM help in assigning appropriate values?\n\nThank you for your question. If we understand correctly, you may be referring to whether STE-MM could be extended to assist with hyperparameter tuning when merging models with distinct weights. In its current form, STE-MM does not explicitly address hyperparameter optimization—its objective is to discover effective permutations for merging models.\nThe algorithm itself only includes standard training hyperparameters such as learning rate and batch size. For example, in Equation 5, the values of $\\lambda_1, \\ldots, \\lambda_n$ are drawn from a uniform distribution and are not considered hyperparameters. Therefore, applying STE-MM to hyperparameter selection would require substantial modification, as this lies outside its intended scope.", "claims_and_evidence": "The authors claim that previous methods fail to achieve linear mode connectivity among more than three models. This is demonstrated in the experiments in Section 3, where, even when the remaining models are transformed into one specific model—thus theoretically entering the loss basin of that model—merging these models still leads to an increase in loss and a decrease in accuracy.\n\nThe authors propose a new search method, STE-MM, to efficiently transfer multiple models into a single loss basin using a permutation matrix. They conduct experiments on MLP, VGG-11, and ResNet-20 trained on MNIST, FMNIST, and CIFAR-10, comparing their method with MergeMany and CCMM. The results indicate that: (1) the acceleration method introduced in Section 4.2 speeds up the search process; (2) their method generally outperforms others in terms of loss evaluation and accuracy on benchmark datasets; (3) they define the sharpness of the loss at the center point and demonstrate that it decreases.", "methods_and_evaluation_criteria": "The models and evaluation criteria are well-suited for the field of computer vision. Although they do not involve NLP, this is not a limitation in the given context.", "theoretical_claims": "The theoretical claims see the (4) in Supplementary materials.", "experimental_designs_or_analyses": "See Claims and Evidence.", "relation_to_broader_scientific_literature": "The paper proposes a merging method grounded in theoretical principles, which has been empirically demonstrated to be effective for model merging involving multiple models. However, other merging methods, such as TIES merging and DARE, are also widely used and have been shown to be effective. Additionally, due to computational cost considerations, simple averaging remains a common approach in multiple-model merging.", "essential_references_not_discussed": "Most essential related works, as I know, are mentioned.", "other_strengths_and_weaknesses": "S1. The paper writing is quite well and clear.\n\nS2. See **Claims and Evidence.**\n\nW1. See **Relation To Broader Scientific Literature.**", "other_comments_or_suggestions": "- The clarity of Figure 5 could be improved. It appears to illustrate that the distance ratio of STE-MM is more concentrated; however, the overlapping histograms make it difficult to discern specific details. Enhancing the visualization, such as by adjusting transparency or using distinct color schemes, may help improve readability.\n- Typos: 059 (right column) model; 061 (right column) denotes.", "questions_for_authors": "- Could STE-MM also assist in determining the optimal hyperparameters for different models? For instance, in model merging, where different models have distinct weights, can STE-MM help in assigning appropriate values?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sPM8FgKjS7", "reviewer": "Reviewer_sFBh", "summary": "This paper investigates permutation-based methods for merging multiple models. The literature mainly focused on merging pairs of models, and this paper shows that these methods fail to transfer multiple models into the same basin. Then they introduce a method for merging multiple methods and find multiple permutations extending the Stringh Through Estimator method and propose an accelerated version of weight matching for faster permutation search", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful feedback and the effort you put into reviewing our work.\n\n> The algorithm explanation could be improved. I suggest the authors expand and clarify the reason for dummy variables and how the algorithm works in general by following the explanation of the seminal paper (Ainsworth et al 2023), section 3.3.\n\nThank you for your suggestion on improving the explanation. We will revise the description of STE-MM to make it clearer, drawing on the explanation provided in Ainsworth et al. (2023), Section 3.3.\n\n> Experiments are only performed on academic benchmarks such as CIFAR10 and MNIST.\nI’m sorry about this comment, I hope it does not trivialize your analysis, but I recommend testing the scalability of the method on more complex tasks such as ImageNet.\n\nThank you for your suggestion. To evaluate the scalability of our method, we conducted additional experiments of model merging with ResNet-50 models trained on the ImageNet dataset. Due to space limitations in the rebuttal, the detailed experimental results are included in our response to Reviewer SJMK’s first comment. The results demonstrate that our method scales effectively to more complex tasks such as ImageNet.\n\n\n> typos: 190: Permutaion Search for Multiple Models 403 right - experimetnal results\n\nWe are sorry for the typos. We will carefully review the entire paper and make the necessary corrections.\n\n> can the authors provide an analysis of the permutation statistics, for example, what is the percentage of the weights that are being permuted?\n\nYes, certainly. The following table provides an analysis of how closely the permutation matrices discovered by STE-MM resemble the identity matrix. Specifically, we measured this by counting the number of entries equal to 1 in the diagonal of each permutation matrix and dividing this count by the size of the matrix. This matching rate was calculated for each layer, then averaged across all layers in a model, and finally averaged across all models being merged. The table reports the mean and standard deviation over three model merging trials. As shown, the matching ratio remains close to zero regardless of the number of models, indicating that the found permutation matrices are significantly different from the identity matrix.\n\n**Percentage of matches between the permutation matrix and the identity matrix [\\%]**\n\n| #models | MLP, MNIST        | MLP, FMNIST       | VGG-11, CIFAR10   | ResNet-20, CIFAR10   |\n|---:|:------------------|:------------------|:------------------|:---------------------|\n|  2 | $0.174 \\pm 0.099$ | $0.217 \\pm 0.136$ | $0.336 \\pm 0.122$ | $0.217 \\pm 0.033$    |\n|  3 | $0.141 \\pm 0.082$ | $0.228 \\pm 0.117$ | $0.298 \\pm 0.258$ | $0.191 \\pm 0.051$    |\n|  4 | $0.152 \\pm 0.075$ | $0.217 \\pm 0.075$ | $0.254 \\pm 0.126$ | $0.242 \\pm 0.023$    |\n|  5 | $0.157 \\pm 0.052$ | $0.212 \\pm 0.033$ | $0.171 \\pm 0.012$ | $0.203 \\pm 0.002$    |\n|  6 | $0.208 \\pm 0.069$ | $0.152 \\pm 0.015$ | $0.174 \\pm 0.036$ | $0.227 \\pm 0.012$    |\n|  7 | $0.177 \\pm 0.023$ | $0.181 \\pm 0.070$ | $0.138 \\pm 0.043$ | $0.221 \\pm 0.027$    |\n|  8 | $0.158 \\pm 0.052$ | $0.180 \\pm 0.019$ | $0.160 \\pm 0.034$ | $0.193 \\pm 0.006$    |\n|  9 | $0.146 \\pm 0.008$ | $0.217 \\pm 0.012$ | $0.141 \\pm 0.032$ | $0.227 \\pm 0.046$    |\n| 10 | $0.178 \\pm 0.044$ | $0.219 \\pm 0.018$ | $0.176 \\pm 0.068$ | $0.232 \\pm 0.010$    |\n\n> another thing is, can the authors provide a baseline where no permutation is applied? I think it is important to establish that there are actual permutations in the considered setting.\n\nThe following table shows the test accuracy of the merged model when the models are combined without applying any permutation matrices. As shown, the accuracy drops significantly as the number of models being merged increases. This result highlights the importance of searching for appropriate permutation matrices when merging multiple models.\n\n**Test accuracy of the merged model without permutation**\n\n| #models | MLP, MNIST         | MLP, FMNIST        | VGG-11, CIFAR10    | ResNet-20, CIFAR10   |\n|---:|:-------------------|:-------------------|:-------------------|:---------------------|\n|  2 | $82.223 \\pm 5.318$ | $46.930 \\pm 7.659$ | $80.353 \\pm 0.365$ | $86.653 \\pm 1.317$   |\n|  3 | $23.020 \\pm 9.077$ | $14.340 \\pm 2.318$ | $32.777 \\pm 4.380$ | $45.780 \\pm 8.533$   |\n|  4 | $9.760 \\pm 0.017$  | $10.043 \\pm 0.075$ | $10.000 \\pm 0.000$ | $10.267 \\pm 0.281$   |\n|  5 | $9.740 \\pm 0.000$  | $10.003 \\pm 0.006$ | $10.000 \\pm 0.000$ | $10.013 \\pm 0.023$   |\n|  6 | $9.740 \\pm 0.000$  | $10.010 \\pm 0.017$ | $10.000 \\pm 0.000$ | $10.000 \\pm 0.000$   |\n|  7 | $9.740 \\pm 0.000$  | $10.000 \\pm 0.000$ | $10.000 \\pm 0.000$ | $10.000 \\pm 0.000$   |\n|  8 | $9.740 \\pm 0.000$  | $10.000 \\pm 0.000$ | $10.000 \\pm 0.000$ | $10.000 \\pm 0.000$   |\n|  9 | $9.740 \\pm 0.000$  | $10.000 \\pm 0.000$ | $10.000 \\pm 0.000$ | $10.000 \\pm 0.000$   |\n| 10 | $9.740 \\pm 0.000$  | $10.000 \\pm 0.000$ | $10.000 \\pm 0.000$ | $10.000 \\pm 0.000$   |", "claims_and_evidence": "Claims are supported by the experiments.", "methods_and_evaluation_criteria": "The paper is clear and introduces experiments to motivate the necessity of a new method to tackle the problem of merging multiple models. Then the proposed method is benchmarked against the correct set of baselines and it scales much better with the number of merged models in terms of loss.", "theoretical_claims": "There is a theoretical section (proof in appendix) claiming that finding n-1 permutations is sufficient. This result serves as a base for their algorithm 1. The proof seems correct.", "experimental_designs_or_analyses": "The experimental design is sound, all claims are supported by experiments.", "relation_to_broader_scientific_literature": "yes. It connects to seminal papers on linear mode connectivity, permutation-based methods and also recent works on merging multiple models (Crisostomi et al 2024)", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\n- important problem\n- scalable method over the number of models\n- experiments are sound and show improvement over the baselines\n- interesting analysis on the flatness of the final solution\n\nWeaknesses: \n- The algorithm explanation could be improved. I suggest the authors expand and clarify the reason for dummy variables and how the algorithm works in general by following the explanation of the seminal paper (Ainsworth et al 2023), section 3.3.\n\n- Experiments are only performed on academic benchmarks such as CIFAR10 and MNIST. I’m sorry about this comment, I hope it does not trivialize your analysis, but I recommend testing the scalability of the method on more complex tasks such as ImageNet.", "other_comments_or_suggestions": "typos:\n190: Permuta*ion Search for Multiple Models\n403 right - experime*tnal results", "questions_for_authors": "please see weaknesses and strengths, and:\n\n- can the authors provide an analysis of the permutation statistics, for example, what is the percentage of the weights that are being permuted? \n- another thing is, can the authors provide a baseline where no permutation is applied? I think it is important to establish that there are actual permutations in the considered setting.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1on36SzKPf", "reviewer": "Reviewer_SJMK", "summary": "Prior work showed that linear mode connectivity can be achieved between two independently trained neural networks by applying an appropriate parameter permutation, suggesting that SGD-trained models converge to a shared low-loss basin under permutation symmetries. This paper extends their analysis to multiple models, proposing STE-MM, a runtime-optimized permutation search method that maintains low test loss after merging and reduces loss sharpness as the number of models increases, indicating that linear connectivity generalizes to multiple models.\n\n## Update after rebuttal\nI thank the authors for the rebuttal. I maintain my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your valuable comments and for your close reading of the paper.\n\n> I recommend that the authors extend their approach to more challenging datasets, such as CIFAR-100 and ImageNet, to further validate its effectiveness.\n\nThank you for your suggestion. To test our method on more complex tasks, we performed an experiment on model merging using ResNet-50 models trained on the ImageNet dataset. The following table presents the test losses of merged models using various permutation search methods, including STE-MM. Results are shown for up to six models due to GPU memory limitations. The ResNet-50 models were trained using a training script published on GitHub by the FFCV library (i.e., https://github.com/libffcv/ffcv-imagenet). The table reports the average and standard deviation of the test loss across three model merging trials. As seen in the table, the test loss of the merged model using STE-MM decreases monotonically as the number of merged models increases, while that of other permutation search methods increases, demonstrating the effectiveness of our method. Note that the test loss of the merged model could potentially be improved by increasing the model width, even when multiple models are merged. Ainsworth et al. (2023), for instance, have shown that the test accuracy of the merged model improves with increased model width when two ResNet-50 models are merged.\n\n**Test loss of the merged model when ResNet-50 models trained on ImageNet are combined**\n\n| #models | MergeMany         | CCMM              | STE-MM            |\n|---:|:------------------|:------------------|:------------------|\n|  2 | $5.835 \\pm 0.134$ | $5.822 \\pm 0.066$ | $5.306 \\pm 0.022$ |\n|  3 | $6.341 \\pm 0.023$ | $6.399 \\pm 0.062$ | $5.179 \\pm 0.044$ |\n|  4 | $6.555 \\pm 0.034$ | $6.576 \\pm 0.047$ | $4.887 \\pm 0.039$ |\n|  5 | $6.661 \\pm 0.084$ | $6.657 \\pm 0.040$ | $4.684 \\pm 0.010$ |\n|  6 | $6.689 \\pm 0.047$ | $6.677 \\pm 0.037$ | $4.559 \\pm 0.027$ |\n\n> Typo \"mdoel\" on line 060, second column\n> Please revise lines 115-134\n> could be shown in a dedicated plot.\n\nWe apologize for any typos, unclear figures, or ambiguous expressions in the text. We will carefully revise the manuscript to address all of these issues in the camera-ready version.\n\n> I suggest to move Fig. 7 and Fig. 8 to the main paper.\n\nThank you for your suggestion. As you mentioned, these are important results, so we will include Figures 7 and 8 in the main body of the paper.\n\n> could it be used to analyze potential biases or inefficiencies in other, more lightweight, permutation search techniques?\n\nThis is a valuable perspective. Since methods based on $L^2$ distance, such as MergeMany and CCMM, do not require training, they can search for permutation matrices with low computational cost. However, our experimental results show that the performance of the merged model deteriorates as the number of models being merged increases when using these methods. This suggests a fundamental difference between the permutation matrices discovered by STE-MM and those identified by lightweight approaches. A more in-depth comparison of these permutation matrices may help uncover the causes of inefficiency in the latter and guide improvements in their design.\n\n> Could you comment on how your paper relates to the concurrent work https://arxiv.org/abs/2403.07968?\n\nThank you for the reference. We were not previously aware of this paper. It proposes the star domain conjecture and introduces the Starlight algorithm to verify it by identifying a parameter $\\theta^\\ast$ from multiple SGD solutions. The authors show empirically that such a $\\theta^\\ast$ exists even for models with smaller widths. Entezari et al.'s conjecture (i.e., multiple SGD solutions can be transferred into a single approximately convex basin via permutations) makes a stronger claim than the star domain conjecture, as shown in the figure on page 2 of their paper.\n\nThis result is particularly interesting, as the method performs well in the small-width regime. In contrast, our STE-MM approach requires sufficiently wide models to transfer them into a shared approximately convex basin; otherwise, a significant barrier remains.\n\nIt is unclear whether this limitation arises from the difficulty of searching the large permutation space—due to the discrete nature of the optimization—or from a more fundamental property of the loss landscape.\n\nIf the latter is the case, it may suggest that wider (i.e., overparameterized) models tend to induce simpler loss landscapes. In fact, a recent theoretical study (https://openreview.net/forum?id=4xWQS2z77v) on two-layer neural networks has shown that increasing the width leads to a simpler structure in the loss landscape. It is plausible that a similar phenomenon occurs in deeper networks as well. We believe that further investigation into this aspect is a promising direction for future research.", "claims_and_evidence": "Yes, I find the results in the paper highly interesting and the work exceptionally well executed. I especially appreciate the approach of optimizing the permutation matrix search by reusing dual variables from previous iterations to dynamically adjust the cost matrix, significantly improving the efficiency of the LAP solver.", "methods_and_evaluation_criteria": "Yes, the proposed approach and insights demonstrate a strong understanding of classical algorithms. The choice of datasets and baselines is appropriate. I recommend that the authors extend their approach to more challenging datasets, such as CIFAR-100 and ImageNet, to further validate its effectiveness.", "theoretical_claims": "The theoretical and algorithmic aspects of the paper are well-explained and appear solid.", "experimental_designs_or_analyses": "The presented results are compelling and should be well-received by the conference community.", "relation_to_broader_scientific_literature": "Both the proposed optimization method (STE-MM) and the findings presented in the paper make a significant contribution to the scientific literature on validating the permutation hypothesis.", "essential_references_not_discussed": "The references are appropriate.", "other_strengths_and_weaknesses": "The paper presents a valuable result that advances the theoretical understanding of deep learning and introduces a method that is highly relevant for applications such as multi-task and distributed learning. I don't see major weaknesses.", "other_comments_or_suggestions": "- Typo \"mdoel\" on line 060, second column\n- Please revise lines 115-134 in the second column to ensure that the referenced conjecture is clearly presented as a conjecture rather than a theorem.\n- Line 350: \"While the loss of the merged model also tends to in crease with the number of models in STE-MM, the amount of increase in test loss value becomes small as the number of models grows.\" - I think it's hard to see this in the provided figures, but could be shown in a dedicated plot.\n- I suggest to move Fig. 7 and Fig. 8 to the main paper - these are quite essential.", "questions_for_authors": "- Since STE-MM is the first method to empirically place multiple models into a single basin, could it be used to analyze potential biases or inefficiencies in other, more lightweight, permutation search techniques?\n\n- Could you comment on how your paper relates to the concurrent work https://arxiv.org/abs/2403.07968?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5II4QWb11B", "reviewer": "Reviewer_K8ki", "summary": "The authors observe the linear mode connectivity hypothesis as proposed before has only been confirmed between two independent models, and propose an algorithm to merge multiple models such that the test loss doesn’t meaningfully change but the model has an approximately flat global minima.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful comments and for taking the time to carefully read our paper.\n\n> The contribution seems important for the larger literature on linear mode connectivity, but this isn't particularly well motivated in the paper itself. The STE-MM algorithm seems novel, but because the methods either lower test accuracy or provide a very small increase, my understanding is that these algorithms are not intended to improve metrics for their own sake but to give more evidence that linear mode connectivity of independently trained networks is empirically true. I think this requires a bit more discussion in the paper. In particular, the fact that the convex basin of merging multiple models becomes less sharp is intuitive, but the consequences of this fact aren’t elaborated on much.\n\nThank you for your thoughtful feedback. As you pointed out, we believe that our work contributes to the broader understanding of linear mode connectivity (LMC), particularly in the context of merging multiple independently trained models. While we touch on this connection in Section 1 of the paper, it may not have been sufficiently emphasized. To better reflect this perspective, we are considering revising the introduction to explicitly highlight the relevance to LMC and potentially modifying the title to more directly reflect this connection, for example, “Linear Mode Connectivity for Multiple Models Modulo Permutation Symmetries.”\n\nWe use average-case sharpness as a metric because, as discussed in Andriushchenko et al. (2023), it quantifies how much the loss increases under Gaussian perturbations of model parameters. If multiple models can be transferred via permutations into the same convex basin, we expect the loss around their midpoint to remain relatively flat—reflected in low average-case sharpness.\n\nIndeed, our experiments show that STE-MM consistently reduces average-case sharpness around merged models. While this does not provide direct evidence of a single convex basin, it supports the possibility that such a basin exists. A more fine-grained analysis of this phenomenon remains an important direction for future research, which we intend to pursue.", "claims_and_evidence": "The claims seem to be well-supported empirically, that the merging of two models doesn't scale and this new method does.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The contribution seems important for the larger literature on linear mode connectivity, but this isn't particularly well motivated in the paper itself.  The STE-MM algorithm seems novel, but because the methods either lower test accuracy or provide a very small increase, my understanding is that these algorithms are not intended to improve metrics for their own sake but to give more evidence that linear mode connectivity of independently trained networks is empirically true.  I think this requires a bit more discussion in the paper.  In particular, the fact that the convex basin of merging multiple models becomes less sharp is intuitive, but the consequences of this fact aren’t elaborated on much.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper addresses the challenge of achieving linear mode connectivity (LMC) among multiple neural networks (NNs) trained using SGD. The prior works experience degradation in performance when merging > 2 models. The current work introduces a novel method, Straight-Through Estimator for Multiple Models (STE-MM), that directly optimizes for loss barrier. The authors also provide a method to accelerate the permutation search. The proposed method outperforms existing approaches in improving LMC, demonstrated through experiments on multiple datasets and network architectures. 

Overall, the reviewers are in consensus that \
(a) the paper tackles a relevant problem and addresses the limitation of prior work in LMC\
(b) the proposed method STE-MM is novel and constitutes a significant contribution. The reviewers appreciate the optimization approach. \
(c) The empirical evaluations demonstrate the effectiveness of the method. Experiment design is sound. Authors also provide results from ImageNet in rebuttal. \
(d) Paper is well written.

I would recommend a solid Accept with following improvements as pointed by the reviewers \
(1) Improving clarity in Fig 5 by revision\
(2) Proofread and correct typos as pointed by reviewers\
(3) Please revise the title to highlight relevant to LMC\
(4) If possible, moving Fig 7 & 8 to main body
Decision: Accept (poster)