Analysis of Linear Mode Connectivity via Permutation-Based Weight Matching: With Insights into Other Permutation Search Methods
Authors: Akira Ito
Masanori Yamada
Atsutoshi Kumagai
Venue: ICLR 2025 Poster (2025)
Keywords: linear mode connectivity
deep learning
permutation symmetry
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We analyze why the linear mode connectivity of neural networks can be satisfied by permutation-based weight matching.

Abstract: Recently, Ainsworth et al. (2023) showed that using weight matching (WM) to minimize the $L^2$ distance in a permutation search of model parameters effectively identifies permutations that satisfy linear mode connectivity (LMC), where the loss along a linear path between two independently trained models with different seeds remains nearly constant. This paper analyzes LMC using WM, which is useful for understanding stochastic gradient descent's effectiveness and its application in areas like model merging. We first empirically show that permutations found by WM do not significantly reduce the $L^2$ distance between two models, and the occurrence of LMC is not merely due to distance reduction by WM itself. We then demonstrate that permutations can change the directions of the singular vectors, but not the singular values, of the weight matrices in each layer. This finding shows that permutations found by WM primarily align the directions of singular vectors associated with large singular values across models. This alignment brings the singular vectors with large singular values, which determine the model's functionality, closer between the original and merged models, allowing the merged model to retain functionality similar to the original models, thereby satisfying LMC. This paper also analyzes activation matching (AM) in terms of singular vectors and finds that the principle of AM is likely the same as that of WM. Finally, we analyze the difference between WM and the straight-through estimator (STE), a dataset-dependent permutation search method, and show that WM can be more advantageous than STE in achieving LMC among three or more models.

Published as a conference paper at ICLR 2025
ANALYSIS OF LINEAR MODE CONNECTIVITY VIA PERMUTATION-BASER WEIGHT MATCHING: WITH IN- SIGHTS INTO OTHER PERMUTATION SEARCH METHODS
Akira Ito', Masanori Yamada* & Atsutoshi Kumagai"
1 NTT Social Informatics Laboratories 2TTT Computer and Data Science Laboratories lakira.iton masanori. yamada, atsutoshi. kumagai Jnnt:: com
ABSTRACT
Recently, Ainsworth al. (2023) showed that using weight matching (WM) minimize the I distance in permutation search of model parameters effectively identifies permutations that satisfy linear mode connectivity (LMC), where the loss along a linear path between two independently trained models with different seeds remains nearly constant. This paper analyzes LMC using WM, which is useful for understanding stochastic gradient descent's effectiveness and its application in areas like model merging. We first empirically show that permutations found by WM do not significantly reduce the 1.2 distance between two models, and the occurrence of LMC is not merely due to distance reduction by WM itself. We then demonstrate that permutations can change the directions of the singular vectors, but not the singular values, of the weight matrices in each layer. This finding shows that permutations found by WM primarily align the directions of singular vectors associated with large singular values across models. This alignment brings the singular vectors with large singular values, which determine the model's functionality, closer between the original and merged models, allowing the merged model to retain functionality similar to the original models, thereby satisfying LMC. This paper also analyzes activation matching (AM) in terms of singular vectors and finds that the principle of AM is likely the same as that of WM. Finally, we analyze the difference between WM and the straight-through estimator (STE), a dataset-dependent permutation search method, and show that WM can be more advantageous than STE in achieving LMC among three or more models.
INTRODUCTION
Large-scale neural networks (NNs) are widely used in various fields (Vaswani al., 2017; den Oord et al., 2016; Zhao et al., 2023), and optimizing their parameters poses a massive non-convex optimization problem. Remarkably, stochastic gradient descent (SGD), which is widely used for training NNs, is known to find good solutions despite its simplicity. One hypothesis for this seemingly counterintuitive phenomenon is that the landscape of the loss function may much simpler than previously thought. Several studies (Garipov et al., 2018; Draxler et al., 2018; Freeman & Bruna, 2017) have found that different NN solutions can be connected by simple nonlinear paths with almost no increase in loss. Recently, Entezari et al. (2022) conjectured that Conjecture 1.1 holds, considering all possible permutation symmetries of NNs:
Conjecture 1.1 (Permutation invariance, informal). Let O2 and O, be two SGD solutions (model parameters). Then, with high probability, there exists a permutation π such that the barrier (defined in Definition 2.1) between O2 and π (0,) is sufficiently small.
Here, the barrier represents the increase in loss observed when linearly interpolating between the weights of the two models. If the barrier between two models sufficiently small, we say that linear mode connectivity (LMC) is satisfied between them (Frankle et al., 2020). Conjecture 1.1 suggests that most SGD solutions can be transferred into the same loss basin using permutations. Indeed, some studies (Ainsworth et al., 2023; Singh & Jaggi, 2020) have experimentally demonstrated that this conjecture is valid for various datasets and models using weight matching (WM), which identifies permutations that minimize the 1,2 distance between the weights of two models.

---

Published as a conference paper at ICLR 2025
Understanding LMC principles based permutation symmetries is important not only for compre- hending how SGD works in deep learning but also for its application in model merging (Singh & Jaggi, 2020), where two independently trained models are combined. The method of finding permuta- tions using only the 1,2 distance is particularly versatile, dataset-independent, and computationally efficient. In fact, several studies (Singh & Jaggi, 2020; Wang al., 2020; Guerrero- et al., 2023) have proposed applications of permutation symmetries in model merging, federated learning, and continual learning.
The current theoretical analysis of LMC relies on the feasibility of closely matching NN weights through permutations. Recently, Zhou al. (2023) proved that if the distance between the weights of two models can be sufficiently reduced via permutation. then LMC holds. Intuitively, for two SGD solutions 0, and 0b. if 0, x x (0.) holds for a permutation æ, the outputs of the interpolated model will closely approximate those of the original models 0₂ and 0s.
However, our analysis reveals that even if LMC holds, the permutations found by WM do not significantly reduce the distance between the two models (at most about a 20% reduction). This suggests that LMC is satisfied even when WM does not bring the two models very close (i.e., Og &6 (O,)). Accordingly, this paper seeks to uncover more fundamental reason why LMC holds through the permutations found by WM. Specifically, we demonstrate that singular vectors with large singular values of each weight in the models play crucial role in LMC. Our analysis not only reveals the principle behind WM but also shows that WM may be more advantageous in merging more than two models compared to other methods such as STE.
The contributions of this paper are threefold:
1. Demonstrating that the L distance reduced by WM is not the direct cause of LMC. We empirically show that permutations found by WM do not significantly reduce the L distance between the two models. Our results show that, even when LMC is satisfied, permutations reduce the model weight distance by no more than 20%. Supported by a Taylor approximation, our findings suggest that reducing the 2 distance through permutations is not the direct reason for LMC satisfaction.
2. Revealing the reason why WM and activation matching (AM) satisfy LMC. We analyze WM from the perspective of the function of each layer of the model. Specifically, provide evidence that WM satisfies LMC by aligning the directions of singular vectors with large singular values in each layer's weights. This alignment ensures that the singular vectors with large singular values, which determine the functionality of each layer, become similar between the merged and original models. Additionally, we show that, from the perspective of the input distribution each hidden layer, aligning singular vectors with large singular values can efficiently approximate the functionality between two models, even if the 2 distance cannot be significantly reduced. As a result, the merged model retains functionality similar to the original models, which facilitates LMC. We also conducted experiments with AM and found that the reason why LMC holds in AM is likely the same as in WM.
3. Revealing STE is fundamentally different from WM in principle, which leads to a significant difference between them when merging multiple models. To distinguish WM from other permu- tation search methods that are independent of I,2 distance, we examine the straight-through estimator (STE), which focuses on minimizing the barrier itself rather than the I,2 distance. Our experiments reveal that the permutations found by STE do not align the directions of singular vectors, which is critical difference compared to WM in achieving LMC. Furthermore, we demonstrate experimentally that this difference significantly impacts the satisfaction of LMC among three more models.
2 BACKGROUND AND PRELIMINARIES
2.1 NOTATION
For any natural number & E N, let [k] (1.2. k]. Bold uppercase variables represent tensors, including matrices (c.g..X0 X), and bold lowercase variables (c.g.. x) represent vectors. For any tensor X, its vectorization is denoted by (xc, and |X|| denotes its Frobenius (T.2) norm.
2

---

Published as a conference paper at ICLR 2025
2.2 PERMUTATION INVARIANCE
We consider multilayer perceptrons (MLPs) f(; 0) with L layers for simplicity while our analyses in this paper be applied any model architectures. Here,
RRd is the input the NN, and E R Rupas represents the model parameters, where dma E is the dimension of the input, and dparam EN the dimension of the parameters. Let æς be the output of the E-th layer (i.e., 2g æ. and, for all E [L], 2 = o (W + by]). Here, denotes the activation function, and We and by represent the weight and bias of the f-th layer, respectively. Note that in this MLP, we have
= || ( be), where represents the concatenation of vectors.
NNs have permutation symmetries of weight space. Considering an NN with model parameters 0, for its Ł-th layer,
P Pz: P" P (PW + Pb;) holds, where P is permutation matrix. Note that permutation matrices are orthogonal, so we have P p-1 Therefore, by permuting the input of the (e + s)t layer with PT, the model parameters can be changed without altering the inpul-output relationship of the NN. Specifically, the new weights and bias are given by W PW,, b/ Pby, W = W++1 WT. Such permutations can applied all layers. We denote the tuple of permutations corresponding to each layer as = (P:) Moreover, if model f is given, the application of permutation
is denoted by (0)).
2.3 LINEAR MODE CONNECTIVITY (LMC)
Let 0 E param be model and L(0) denote the value of the loss function for the model 0. Here, we define the loss barrier between two given models O2 and O, as follows:
Definition 2.1. For two given models 0₂ and 0,, their loss barrier is defined as
B(0., 0s) = max (C(20, + +1 (1 - A)0,) - (..( + - ())))()) X
Intuitively, the barrier represents the increase in loss due to the linear interpolation of the two models. Two models O, and 0g are said to be linearly mode connected if their loss barrier is approximately zero.
2.4 PERMUTATION SELECTION
Entezari et al. (2022) conjectured that for SGD solutions 02 and 0g, there exists permutation ı such that LMC holds between 0, and (0,) with high probability. Afterward, Ainsworth et al. (2023) proposed WM, straight-through estimator (STE), and activation matching (AM) as methods for finding such permutations. This subsection explains WM, which is the main focus of this paper. AM and STE discussed in Sections 5 and 6.
In WM, we search for a permutation that minimizes the L diistance between two models!:
arg min 00,- T (0,) arg min 3
Σ [w P.W. ( PL
x
EE(L]
where, without loss of generality, let Pt. I and Po I, and is an identity matrix. This minimization problem is known as the sum of the bilinear assignments problem. which is NP- hard (Koopmans Beckmann, 1957; Sahni & Gonzalez, 1976; Ainsworth et al., 2023). Recently. Guerrero- -Para et al. (2023) proposed solving Equation (1) using Sinkhorn's 's algorithm (Adams & Zemel, 201 1) by using an optimal transport problem. We adopt their method because allows for the optimization of all layers simultaneously. unlike the method by Ainsworth et al. (2023), and potentially finds better solutions.
MOTIVATING OBSERVATIONS
Previous studies have suggested that the closeness of two parameters terms of I distance is important for satisfying LMC. For example, Zhou et al. (2023) showed that LMC holds if
Athhoggh only the weights are considered here, the biases can also be dealt with by concatenating the biases and the weights.
3

---

Published as a conference paper at ICLR 2025
Table 1: Results of WM and the estimated barrier value using Taylor approximation when À 1/2. The table presents the mean and standard deviation from five trials of model merging (i.e., the linear combination of the models (0, + =(0))/ The columns labeled "Barrier". "Taylor approx.", and "Diff." show the barrier value, the estimated barrier value using Equation (2) for the merged model at = 1/2, and their difference, respectively. In the "Diff." column, if a statistical significant difference is determined using a I-test at 5% significance level, they are highlighted in bold. The table also shows the L distance between the models 6, and 0, before and after applying the permutation, as well as the reduction rate of the L distance (i.e., (1) Poll - 0. TT (0))) 1)) 0b|l).
Danasct
Network
Barrier (A 1/2)
Taylor approx.
Dift.
0. 0.1
0, [0,)
Reduction rate [%)
CIFARIO
VGGU
0.035 # 0.1
2.956 + 0.35
2.921
0.323
799 503 + 16.399
746
19.576
6.64 + 0.808
ResNer20
0.167 + 0.005
7.517 0.573
7.349
0.599
710.702 16.201
005 12.530
6.987 0.472
FSINSST
MLP
-0.183 = 1.039
0.928
= 0.7
1111
0.152
121 153 5.83
1000111 2 4.71
17.897 =: 0.33
MNIST
MLP
0.033 0:006
0.036 a 0:00
0.069
0.028
81.231 5.58
64.751
4.795
20.305 1: 1.225
commutativity property is satisfied. This property holds if, for all layers E, W P.WW P._ P =0. Zhou et al. (2023) argued in Section 5.2 that since WM finds the permutation that minimizes Equation (1), WM can be seen as searching for permutations that satisfy the commutativity property. In particular, there is huge number of permutations because the total number of possible permutations grows exponentially as the number of layers and the width increase, and thus, some of them may sufficiently reduce the distance between the two models. However, this section explains from the perspective of Taylor approximation that this intuition is not always correct. Our results demonstrate that even when LMC is satisfied, the permutations found by WM do not necessarily bring the models as close as expected. The facts observed from the experiments in this section motivate us to explore other reasons for satisfying LMC in the following sections.
3.1 CLOSENESS OF TWo MODELS IN TERMS OF TAYLOR APPROXIMATION
This subsection describes the estimation of the barrier value using the Taylor approximation. Let 02 and 0, be two SGD solutions, and π be a permutation found by WM to make π (0,) close to the model O.. Let O.- ^O.- + (1 X)TT (O,) be the merged model at a ratio À E (0, 1). If O. and (0,)) are sufficiently close, then their linear interpolation 0. should be close to both models. Therefore, the loss of the parameter O. should be able to be approximated by the Taylor approximation. In fact, the following theorem holds if 6. and π (0,) are sufficiently close:
Theorem 3.1. The loss function L U 3 0 L(0) E R is assumed to be of class C3 on an open set U over Rdyan,, Let Ha and HA be the Hessian matrices centered at the models O2 and π (0,), respectively. for any X E (0, 1), A0, + (1 x)0, E U holds, then we have
T(.)) A \(1 T
2 8 TN ((1-)))) AHb
(2)
where V is the gradient with respect to the parameters, O is the Landau symbol, B is the L2 distance between O2 and T0,)), and μ is the unit vector from O2 to x (0,)) (i.e., a (T(()) 0 0a)/3).
We prove this theorem in Appendix G.I. The theorem states that if 9. and (0,) are sufficiently close, the barrier value can be predicted from the gradients and Hessian matrices around each model.
3.2 EXPERIMENTAL RESULTS
We conducted experiments to verify whether the Taylor approximation (Equation (2)) accurately estimates the barrier between two SGD solutions. Table 1 presents the experimental results of model merging. Details about the datasets, network training procedures, and permutation search methods used in these experiments are described in Appendix D. In the table, we chose X = 1/2 because it is empirically known that the midpoint between two models results in the highest loss (Ainsworth et al., 2023; Guerrero-Pene et al., 2023). It is worth noting that Adilova et al. (2024) demonstrated that the location of the highest barrier shifts when one model is more generalized than the other, although not directly applicable to this paper as the two models are generalized to the same extent. We used the
4

---

Published as a conference paper at ICLR 2025
Singular value
MLP, MNIST
Singular value
MLE FMNIST
Singular value
VGGL 1, CIFAR10
Singular value
ResNet20, CIFARIO
10-
10
40-
20
0
20000 40000 60000
100000 200000
Index
Index
Index
Index
Figure 1: Distribution of the singular values in the second layer. The singular values of ten in- dependently trained models (i.e., trained with different seeds) are plotted in different colors. The distribution of the singular values for all layers shown in Appendix H.2.
vhp function provided in the PyTroch library² efficiently compute μ H, which is required for the evaluation of Equation (2). A negative value in the Barrier column indicates that the loss of the merged model is lower than those of the original (pre-merge rere-mereed models.
The table shows that for all datasets, there is a significant difference between the actual barrier values and those estimated by the Taylor approximation. These differences are particularly large for VGG11 and ResNet20. Additionally, the table indicates that the L distance changes by only about 6% to 20% from the original distance. This suggests that WM does not bring the models sufficiently close, at least not close enough for a second-order Taylor approximation to hold.
ANALYSIS OF WM
The previous section demonstrates that the establishment of LMC by WM not due to the reduction in L distance itself, but rather because WM helps find permutations that result in smaller barrier between the two models. To better understand why WM reduces the barrier, we first analyze WM by performing SVD on the weights of cach layer of the model in Section 4.1. Then, in Section 4.2, show that the singular value distribution of each layer is almost identical across independently trained models, and that the primary differences between the models vectors. In Section 4.3, we demonstrate that WM preferentially aligns the directions of singular vectors with large singular values between the weights of the two models. Finally, in Section 4.4, we explain that aligning singular vectors with large singular values makes LMC more achievable because these singular vectors predominantly influence the outputs of the hidden layers of the models.
due to variations in their singular
4.1 ANALYSIS BASED ON SVD
The basic idea in analyzing WM is to perform SVD the weight in each layer. Although using SVD for analysis might seem overly simplistic given that WM reduces the L distance, this approach
provides important insights that are explored in subsequent sections. In WM, permutation matrices
are searched to minimize Equation (1). We denote the SVDs of ((()) and w.(h)
g by W (a)
Ul") St ((a) (a)) E, uz 86, (ut,))
(a) (a) (T and w((h) U S (v:h) = E, u u (v2
(b) (b)
(b)
respectively. Here, assume that the singular values ordered in descending order (i.e., for all
l E [T.. s 2 $22 2 .... 2 s and (b)
(a)
(e)
F 2 8 2 2 i,n' where the number of singular
(b)
values). Then, we have
arg argmin|| O (0.))l = arg min
Σ u
Pru (0) (8) T| (P-
T|p
(3)
EELL]
Equation (3) shows that the permutation matrices P7 and P are multiplied by the left and right singular vectors of the model 6,. respectively. The L distance between the models is expressed by the difference in singular values and singular vectors between the two models, as indicated by Equation (3). Therefore, in the following, we will discuss the differences in (1) singular values and (2) singular vectors of independently trained models.
functional..
html
5

---

Published as a conference paper at ICLR 2025
R(B. m.mm
0.4
wo WM
0.2
V-0.3
y-0 MNIST
V-0.3 y=0 MLP FMNIST
y-0.3 VGG11, CIBARTO
y-0
y 0.3 ResNet20, CIFARI0
y-0
Figure 2: Mean and standard deviation of 2( 0₂) from five permutation searches using WM. The red and blue bars represent the results with and without applying permutation to 0,, respectively.
4.2 DIFFERENCES BETWEEN SINGULAR VALUES OF TWO MODELS
First, investigate the differences between the singular values of two independently trained models. To this end, ten models are trained independently under identical conditions except for the seed, and their singular values are compared. Figure plots the singular values in the second layer of independently trained models in descending order. The evaluation results for all the layers are shown in Figure 7. As can be seen the figures, in the hidden layers, the singular values are very close across all models. Therefore, the differences in singular values between the models are not significant obstacle to reducing the distance between the two models zero.
4.3 SINGULAR- VECTOR ALIGNMENT
In the previous subsection, confirmed that the distributions of the singular values of the weights of the two independently trained models were almost equal. In this subsection, we will show that the permutations found by WM preferentially align the dominant singular vectors of the two models, and cannot align all singular vectors between the two models. Therefore, WM cannot reduce the L distance to zero.
First, we introduce the following theorem:
Theorem 4.1. Given the trained L-layer MLPs Oa and 0g, Equation (3) equivalent
arg min |0, -r0))) = arg max (Pec E,1
(a))
(4)
x
The proof of this theorem is shown in Appendix G.2. Focusing the term for each layer (a) (b)
Eng St, St.j (u)))) ( (P.ul))
(T (P:..i in Equation (4), (u()) (P:u (b)) is the inner prod- uct between the left singular vector (0) of the model 6, and the left singular vector u of the model 0, applied with the permutation matrix Pt. The permutation matrix orthogonal, so it only permutes the elements without changing the norms of the left singular vectors. Therefore, this inner product maximized when the directions of the two left singular vectors are aligned by the permutation matrix. The same applies to the right singular vectors. Thus, Equation (4) can be interpreted as finding permutation matrices that align the directions of the singular vectors for all layers between two models, especially those associated with large singular values. Like MLPs, Appendix shows similar analysis holds for convolutional layers.
Then, empirically evaluate how well the singular vectors aligned, we calculate
Pru (u))
(a))
(u(.))
(a)
Po-10( (b)
R(O,, T (0.))
Enne
where ny is the number of singular values in the f-th layer. Note that R n (0x)) S holds and, (a) we have equality if for all l and i, u Pau (6) and 02 1o (proof in Appendix G.5). Therefore, if R(O., π (6,)) is close to one, the singular vectors of the models well-aligned.
(.())
(b)
Figure 2 shows the experimental results evaluating the R value between 6, and ((e)). A threshold y is introduced to examine whether the singular vectors with large singular values preferentially aligned. For each model, we evaluate using only singular vectors whose ratio to the largest singular value is greater than 7. Thus, in the figure, 7 0 corresponds to the results when all singular vectors

---

Published as a conference paper at ICLR 2025
1.0
1.
68-
68-
6.6
10
MLB MNIST
UMNNSTT
VGG813, CIPARDO
vil RacNieta CIRARIO
WM MNBIT
WM MLB PNNNST
CIPARI
Rechiet2 cunaato
(a) Evaluation results with the threshold y 0.
(b) Evaluation results with the threshold 0.3.
Figure 3: Evaluation results of R between the pre- and post-merged models. The red and blue bars respectcii respectively. represent the evaluation results of R(O.,, (0.+)(0)))/2 and R(T(Os), (0.+-T (0.)))
are used, and 7 = 0.3 corresponds to the results when only singular vectors with a ratio to the largest singular value exceeding 0.3 are used. When we calculated R, its denominator Ee ne also adjusted according to the value of y. The details of the calculation of R are described in Appendix B.
The figure shows that the directions of the singular vectors aligned with WM. Without WM, the value of R is almost zero, indicating that the singular vectors are nearly orthogonal. Additionally, focusing on the difference in y, when the singular vectors are aligned using WM, the value of R clearly larger when ry is 0.3. This indicates that WM aligns singular vectors with larger singular values more closely. Although the value of R is not necessarily very large, especially around 0.2 most for VGGII and ResNet20, this alignment of singular vectors still affects the merged models.
Figure shows the evaluation results of R between the merged model (i.e., (0a n (0,)). and the pre-merged models (i.e.. 0₂ and (0.)). To investigate how well the directions of singular vectors with large values are aligned between the merged and pre-merged models, we also show the results for y = 0.3 in Figure 3(b). The figures show that when regardless of the use of WM. However, when y 0.3, the value of R changes significantly depending on whether WM is used. For example, the MLP results show that the value of I exceeds 0.8 when using WM. This result indicates that the directions singular vectors with particularly large singular values are better aligned between these models.
O. the value of R does not change
4.4 IMPORTANCE OF SINGULAR VECTORS LMC
In the previous section, we mentioned that WM aligns the directions of singular vectors with large singular values. This section clarifies why these singular vectors, rather than L distances, play crucial role in establishing LMC.
To explain this, we first focus on the difference between outputs the f-th layers of two models, (( and ( given the same input (e.g., = (a) or = = 2 Suppose that the distributions of the singular values of the two weights equal (this assumption holds for models trained with SGD, as shown in Figure 7). The difference between the outputs can be bounded from above:
(W (a) =) (W:") ( 2)1l S CE ( (o) W (b) 2||:
(5) where is a Lipschitz continuous activation function with constant C 0 (e.g., C 1 for the ReLU function). From Equation (5), we can see that depending on the distribution of the input the outputs of the layers can be close even when the distance between the two weights is not.
Let w(()) = Σ, uis
(b))T T be the SVDs of their weights. Here, we assume that, for some index k,, the direction of z is always in the direction of the k-
(a)
(w(.)) ( T and W!()) u
th right singular vector 22 with W(a) Then, the product between w(() and z is given by w(()) = E. u s, (v Therefore, Equation (5) can be rewritten
(a)
(a) (a) (a) )
(a) (a) ( because the singular vectors are orthogonal. u (v(a))
Ello (a) (W? 2)1l < CT ue
Σ u
2
Ue
Thus, as long as the directions of the k-th singular vectors of the two weights are aligned (i.e., (a) (b)
0ikk -2.6 and u uz.k hold), the outputs of the layers will coincide regardless of the other
(b)
7

---

Published as a conference paper at ICLR 2025
VGG11, CIFARIO
ResNet20, CIFARIO
MLP, MNIST
MLP FMNIST
1e6
1e7
Elyiz|
10
1.0
1.0
5-
0.5
0.5
0.0
0.0
200
400
200
400
20000
40000
60000
100000 200000
Singular Vector Index
Singular Vector Index
Singular Vector Index
Singular Vector Index
Figure 4: Average absolute values of the inner products of the right singular vectors and the input of the second layer. The figure shows results for ten models trained with different seeds, each represented by different color. The test dataset is used as input for the models. In each plot, the vertical axis denotes the value of =e- and the horizontal axis denotes the index of the right singular vector. The left side of each plot corresponds to singular vectors with large singular values. The results for all layers are shown in Figure 9.
singular vectors. Note that the L distance between the two weights is not necessarily close to zero since the directions of the other singular vectors need not be aligned. In fact, in Appendix C, we provide example where the I distance is not close to zero even though the output is zero.
More generally, the following theorem holds for the difference between the outputs of the two layers. Theorem 4.2. For the difference, we have
Bllo(W( () 2)11 sc c
Σ
(B(()))
2) +
Σ 18 ( (E((
T2))
2)²
1/2
(( (0)) x(18))
2
E.3 o2
(6)
The proof of Theorem 4.2 is provided in Appendix G.6. If the right- hant- side of Equation (6) in the theorem is small, the difference is also small. Note that each sum on the right-hand side includes the inner product between the right singular vector and the input (i.e., (v z and ( 2). Since singular vectors are orthogonal each other, if is aligned with one singular vector, its inner product
with the other singular vectors will be small. In other words, right singular vectors with large inner product with the input determine the difference between the outputs of the two layers.
In the context of WM, it is desirable that the input vector has the same direction the right singular vectors with large singular values because WM preferentially aligns these singular vectors of the two weights. To verify this, we experimentally investigate the relationship between the directions of the right singular vectors and that of the input vector. Figure shows the value of E(v,,, for the i-th right singular vector 8y,; the second layer (i.e., & = 2) and the corresponding hidden layer input z for each model. The results show that in the hidden layer, the singular vectors with large singular values have large inner products with the input vectors, which indicates that the permutations found by WM make LMC more feasible. 3 In particular, Figure 3(b) shows that by aligning the directions of the singular vectors between the two models 9a and O, using WM, the directions of the singular vectors with large singular values of the models before and after merging (e.g., O, and (0.. + 0b)/2) 2) are well aligned. This suggests that the hidden layer outputs of the models before and after merging are closer, which contributes to the establishment of LMC.
Some studies (Ainsworth et al., 2023; Entezari et al., 2022) have observed that increasing the model width makes it easier satisfy LMC using WM. In Appendix H.4, we provide an empirical analysis to explain this observation in terms of singular-vector alignment. Furthermore, Qu & Horvath (2024) showed that strengthening weight decay and increasing the learning rate makes it easier for LMC to be established through WM. Appendix H.5 demonstrates empirically that increasing these values reduces the proportion of large singular values in the weights of each layer, facilitating the alignment of the corresponding singular vectors through WM, and thus making LMC easier to achieve.
As shown in Figures 11 and 12, this tendency only occurs when the model is sufficiently wide. This suggests that LMC is unlikely be established in WM unless the model wide enough.
8

---

Published as a conference paper at ICLR 2025
Table 2: Results of model merging with STE.
Dataset
Nerwork
Barrier (A 1722)
1. dist. w/o STE
1." dist. w/ W/STE
(00,) - 0.33
CIFARIO
VOGIT
0.06
0.042
739.503
16,396
799.179 = 16.177
0.036 = 0.007
ResNer20
0.119
0.119
710.762
16.261
711.142
16,048
0.013 0.003
IMNIST
MLP
0.342 1 0.066
121.833
5,83
118.316
5,453
0X081 1: 0:008
MNIST
MEF
0.037
0.008
51.231
5.58
73.004
5.58
0.211 0.013
mvle.)
Alm,(Bu). r(l.))
0.4
0.0
FMNIST
VGG11 CIFARIO
ResNe120, CIFAR1O
Figure 5: Evaluation results of R between each pair of the models with y 0.3.
ACTIVATION MATCHING
Ainsworth al. (2023) proposed activation matching (AM) as a permutation search method different from WM. This section compares AM and WM, and explains that their results are almost similar.
AM searches for permutation n based on the following equation:
arg min PPie (E
Bl|z (a) 11
(7)
Unlike WM, AM can be solved as simple linear sum assignment problem because it can be optimized independently for each layer, allowing for the optimal solution to be obtained.
The minimization of Equation (7) is related to Theorem 4.2. Specifically, in a permutation search for the ε-th layer, if we assume that the outputs 2("), and 2 from the previous layer are sufficiently
(b) then minimizing the right-hand side of Equation (6) becomes equivalent to reducing the objective function in Equation (7). In other words, similar WM, AM may search for permutations that align the singular vectors with large singular values between two models. To verify this, the results of model merging using AM are presented in Table The experimental settings are the same as those used for WM Section 3.2. Additionally, to evaluate how well the singular vectors align through permutation, the R calculation results are shown in Figures 15 and 16. These results closely resemble those for WM in Table and Figures and 3, suggesting that the reason AM achieves LMC is likely to be similar to that for WM.
close under the permutation P (i.e., = P 1
6 COMPARISON WITH STRAIGHT-THROULS ESTIMATOR
This section discusses the relationship between the straight-througe estimator (STE), a more direct permutation search method, and WM in terms of singular vectors. STE uses dataset to find permutations with small barrier value. We also explain that STE and WM are based on fundamentally different principles and show how this difference impacts LMC among three more models.
6.1 STRRIGGH ESTIMATOR (STE)
Ainsworth et al. (2023) proposed the STE, which finds a permutation v such that arg min L((O, (0,))) /2).
Since Equation (8) is difficult to solve directly, Ainsworth et al. (2023) proposed a method to approximate the solution. Later, Guerrero-Peña et al. (2023) proposed method to solve Equation (8) directly using Sinkhorm's 's algorithm. We adopt the latter method, which we refer to as STE this paper.
6.2 EXPERIMENTAL RESULTS OF MODEL MERGING BY STE
The experimental results of model merging using STE are shown in Table 2. This table also shows the 7,2 distance and the R value between the models before and after the permutation. Despite

---

Suppose we have three SGD solutions: 0., 0,, and O.. Let m. and т. be permutations that satisfy LMC between 0g and (,)))). and 0n and (.()).. respectively. If permutations found by STE depend on the locality of the loss landscape rather than the linear algebraic properties of the model weights, there is no guarantee that n. (0,) and To(O.) are linearly mode-connected. In contrast, permutations found by WM align the directions of the singular vectors of the two models. This means that the Published as a conference paper at ICLR 2025
Table 3: Loss and accuracy barriers between Th (0,) and (e)).). The table shows the mean and standard deviation over three model merging trials.
Loss barrier ((ms(6.) + 0.1122)
Accuracy ((mx((6)
Dataset
Network
WM
STE
WM
STB
CIFARIO
VGGII
0.141 + 0.141
2.172
0.989
10.12
5.117
32.013
193
ResNer20
0.294 # 0.098
1. 1,693
= 1.168
1.23
= 0.99
31.483
2.416
FMNIST
MLP
-0.174
+ 0.051
0.023
0.
4.337
1.434
15.97 + 1.724
MNIST
MLP
-0.031
1: 0.003
0.017 0.014
0.475
0.069
2.312 1 0.445
the relatively small barrier value, Table 2 shows that the I,2 distance between the two models before and after permutation hardly changes compared to the results with WM shown in Table 1. Since R(O,, π (0,)) is nearly zero, the singular vectors between the two models are likely not aligned at all. Therefore, the reason for satisfying LMC by STE is completely different from that of WM.
6.3 LMC AMONG THREE MODELS
The previous subsection shows that the permutation matrices found by STE do not align the directions of the singular vectors of the models. This suggests that STE finds permutation that reduces the loss of the merged model based on the loss landscape rather than the linear algebraic properties of the weight matrices of each layer. The difference between the principles of STE and WM could result in a qualitative difference in LMC among three or more models.
singular vectors of Ty (0%) and ne (0.) are also expected to be aligned. Thus, the LMC between xz and n.. (0.) may not be satisfied with STE, while it is likely to be satisfied with WM.
We performed model merging experiments among three models to confirm the validity of the above discussion. First, Figure 5 presents the results of examining how well the singular vectors aligned in each model pair by WM. Since the models 0g and 0. are matched to 0₂ through WM, it is expected that R(O,,, To (0,)) and R(Oa, To (0.)) would be large. On the other hand, although O, and O. were not explicitly aligned, (0), (0.)) ) is clearly greater than zero, indicating that the directions of these two singular vectors are indirectly aligned by WM. From this result, the barrier between the models Ty (b,) and (e() ) is expected to be small. To confirm this, Table 3 shows the barriers between n. (0,) and T. (..). Table shows the detailed results, and Figure 17 shows the test accuracy landscape around O no (O,). and (O.)). As can be seen from Table 3, the barrier between n6 (0,) and T. (0.) is smaller with WM than with STE. This means that there is a significant difference between the principles of permutations obtained by WM and STE. Figure 17 also shows that the landscape of test accuracy is flatter around the three models with WM than with STE. Therefore, WM is likely to be more advantageous, especially for merging three or more models.
7 CONCLUSION
This paper analyzed why linear mode connectivity (LMC) is satisfied through permutation search with weight matching (WM). First, we demonstrated that WM does not reduce the distance between the weights of two models significantly as previously thought. We then analyzed WM using singular value decomposition (SVD) and found that WM aligns the directions of singular vectors with large singular values, which plays a crucial role in achieving LMC. Additionally, we showed that the reason LMC is established in AM is likely the same as for WM. Finally, we discussed the difference between STE and WM from the perspective of singular vectors.
Although this paper primarily analyzed WM from the perspective of individual layers (e.g.. Theo- rem 4.2), it remains unclear why our analysis actual network is multi-layered. In the future, a more comprehensive analysis that accounts for the multi-layered structure of the network will be necessary.
explain the phenomenon so effectively since the
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential ethical consequences of our work, none which feel must be specifically highlighted here.
REPRODUCIBILITY
The settings for reproducing the experiments are described in Appendix All the proofs of theorems are given in Appendix G.
REFERENCES
Ryan Prescott Adams and Richard S. Zemel. Ranking via sinkhorn propagation. ArXiv preprint. 2011.
Linara Adilova, Maksym Andriushchenko, Michael Kamp, Asja Fischer, and Martin Jaggi. Layer- wise linear mode connectivity. In Proc. of ICLR, 2024.
Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha S. Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In Proc. of ICLR, 2023.
Jose M. Alvarez and Mathieu Salzmann. Compression-aware training of deep networks. In Proc. of NeurlPS, pp. 856-867, 2017.
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang, Stronger generalization bounds for deep nets a compression approach. Proc. of ICML, pp. 254 203,263345 2018.
Donato Crisostomi, Marco Fumero, Daniele Baieri, Florian Bernard, and Emanuele Rodolà. CRM3; Cycle-consistent multi-model merging. ArXiv preprint, 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In Proc. of CVPR, pp. 248-255, 2009.
Emily L. Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Proc. of NeurIPS, pp. 2014.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred A. Hamprecht, Essentially no barriers in neural network energy landscape. Proc. of ICML, pp. 1308-1317, 2018.
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. In Proc. of ICLR, 2022.
Damien Ferbach, Baptiste Goujaud, Gauthier Gidel, and Aymeric Dieuleveut. Proving linear mode connectivity of neural networks via optimal transport. In Proc. of AISTATS, pp. 3853-3861, 2024.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In Proc. of ICLR, 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In Proc. of pp. 3259-3269, 2020.
C. Daniel Freeman and Joan Bruna. Topology and geometry of half-rectified network optimization. In Proc. of ICLR, 2017.
Tomer Galanti, Zachary S. Siegel, Aparna Gupte, and Tomaso Poggio. Characterizing the implicit bias of regularized sgd in rank minimization, 2022.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin. Dmitry P. Vetrov, and Andrew Gordon Wilson. Loss surfaces, mode connectivity, and fast ensembling dans. In Proc. of NeurIPS, pp. 8803-8812, 2018.
11

---

Published as a conference paper at ICLR 2025
Fidel A. Guerrero-Pen Heitor Rapela Medeiros, Thomas Dubail, Masih Aminbeidokhti, Eric Granger, and Marco Pedersoli. Re-basin via implicit sinkhorn differentiation. In Proc. of CVPR, pp. 20237-20246, 2023.
Anil K. Jain. Fundamentals of digital image processing. Prentice-Hall, Inc., USA, 1989.
Keller Jordan, Hanic Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur. REPAIR: renormalizing permuted activations for interpolation repair. Proc. of ICLR, 2023.
Jakub Konečný, H. Brendan McMahan, Daniel Ramage, and Peter Richtárik. Federated optimization:
Distributed machine learning for on-device intelligence. ArXiv preprint, 2016.
Tjalling C. Koopmans and Martin Beckmann. Assignment problems and the location of economic activities. Econometrica, pp. 53-76, 1957.
A. Krizhevsky and Hinton. Learning multiple layers of features from tiny images. Master's 's thesis, Department of Computer Science, University of Toronoo, 2009.
Rohith Kuditipudi, Xiang Wang, Holden Lee, Yi Zhang. Zhiyuan Li, Wei Hu, Rong Ge, and Sanjeev Arora. Explaining landscape connectivity of low-cost solutions for multilayer nets. Proc. of NeurIPS, pp. 14574-14583, 2019.
Guillaume Leclere, Andrew Ilyas, Logan Engstrom, Sung Min Park, Hadi Salman, and Aleksander Madry. FFCV: accelerating training by removing data bottlenecks. Proc. of CVPR, pp. 12011 12020, 2023.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, pp. 2278-2324, 1998.
Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication--ffiici learning of deep networks from decentralized data. In Proc. of AISTATS, pp. 1273-1282, 2017.
Vaishnavh Nagarajan and J. Zico Kolter. Uniform convergence may unable to explain generaliza- tion in deep learning. In Proc. of NeurlPS, pp. 11611-11622, 2019.
Quynh Nguyen. On connected sublevel sets in deep learning. In Proc. of ICML, pp. 4799, 2019.
Quynh Nguyen, Mahesh Chandra Mukkamala, and Matthias Hein. the loss landscape of class of deep neural networks with no bad local valleys. In Proc. of ICLR, 2019.
Xingyu Qu and Samuel Horvath. Rethinking model re- re-basin and linear mode connectivity, 2024.
Sartaj Sahni and Tcofilo Gonzalez. P-complete approximation problems. J. ACM, pp. 555-565, 1976.
Ruslan Salakhutdinov. Deep learning. In Proc. of KDD, pp. 1973, 2014.
Hanie Sedghi, Vineet Gupta, and Philip M. Long. The singular values of convolutional layers. In Proc. of ICLR, 2019.
Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. In Proc. of NeurlPS, 2020.
Sidak Pal Singh, Linara Adilova, Michael Kamp, Asja Fischer, Bernhard Schölkopf, and Thomas Hofmann. Landscaping linear mode connectivity. In High--imensional Learning Dynamics 2024:
The Emergence of Structure and Reasoning, 2024.
Nadav Timor, Gal Vardi, and Ohad Shamir. Implicit regularization towards rank minimization in relu 2023.
networks. In Proc. ALT, pp.
Murad Tukan, Alaa Maalouf, Matan Weksler, and Dan Feldman. Compressed deep networks: Goodbye svd, hello robust low-rank approximation, 2020.
Aaron den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. ArXiv preprint, 2016.
12

---

Published as a conference paper at ICLR 2025
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. of NeurlPS, pp. 5998-6008, 2017.
Luca Venturi, Afonso S. Bandeira, and Joan Bruna. Spurious valleys in one-hidden-layer neural network optimization landscapes. Journal of Machine Learning Research, pp. 1-34, 2019.
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. Nelson, Eric Jones, Robert Kern, Eric Larson, C Carey, ilhan Polat, Yu Feng. Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, pp. 261-272, 2020.
J von Neumann. Some matrix-inequalities and metrization of matrix-space tomsk univ. rev., 1 (1937), 1962.
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris S. Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. Proc. of ICLR, 2020.
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Korbblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Proc. of ICML, pp. 23965-23998, 2022.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. ArXiv preprint, 2017.
Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. On compressing deep models by low rank and sparse decomposition. In Proc. of CVPR, pp. 67-76, 2017.
David Yunis, Kumar Kshitij Patel, Samuel Wheeler, Pedro Savarese, Gal Vardi, Karen Livescu, Michael Maire, and Matthew R. Walter. Approaching deep learning through the spectral dynamics of weights, 2024.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang. Junjie Zhang. Zican Dong, Yifan Du Chen Yang. Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian- Yun Nie, and Ji-Rong Wen. survey of large language models. ArXiv preprint, 2023.
Zhanpeng Zhou, Yongyi Yang. Xiaojiang Yang. Junchi Yan, and Wei Hu. Going beyond linear mode connectivity: The layerwise linear feature connectivity. In Proc. of NeurlPS, 2023.
13

---

Published as a conference paper at ICLR 2025
APPENDIX
A EXTENDED RELATED WORK
(Linear) mode connectivity. Several studies (Garipov et al., 2018; Draxler et al., 2018; Freeman & Bruna, 2017) have found that different neural network solutions can be connected by nonlinear paths with almost increase in loss. Nagarajan & Kolter (2019) first discovered that solutions can be connected by linear paths with an almost constant loss value when training models on MNIST with the same random initial values. Later, Frankle et al. (2020) demonstrated experimentally that LMC is not always satisfied between two SGD solutions, even with the same initial parameters, depending on the datasets and model architectures. However, they also showed that if single model is trained for certain period and then two models are trained independently from this pre-trained model as a starting point, they are linearly mode-connected. Furthermore, Frankle et al. (2020) explored the relationship between LMC and the lottery-ticket hypothesis (Frankle & Carbin, 2019). Entezari et al. (2022) conjectured that LMC satisfied with a high probability between two SGD solutions by accounting for permutation symmetries in the hidden layers. Subsequently, A Ainsworth et al. (2023) proposed a WM method by formulating neuron alignment as bipartite graph matching problem and solving it approximately. Later, Guerrero-Pene Perr et al. (2023) suggested using Sinkhorn's algorithm to solve the WM directly. Some previous studies (Ainsworth et al., 2023; Crisostomi et al., 2024) have also proposed permutation search methods for achieving LMC between multiple models. However, all of these methods reduce the L distances between the models, and no methods have been proposed that use information the loss function, such as STE. Therefore, in this paper, we created pair of models and performed a permutation search for each pair to clarify the differences between WM and STE in fair manner. The investigation of permutation search methods for multiple models is future work.
While several papers (Venturi al., 2019; Nguyen al., 2019; Nguyen, 2019; Kuditipudi et al., 2019) have discussed nonlinear mode connectivity, there is little theoretical analysis on LMC. Ferbach et al. (2024) provided an upper bound on the minimal width of the hidden layer to satisfy LMC. However, to prove this, they assumed the independence of all neuron's 's weight vectors inside a given layer. It is unlikely that this assumption holds for models after training. Partially similar to our paper, Singh et al. (2024) demonstrated that the barrier value can be approximated using a second-order Taylor approximation for the case of spawning (Zhou et al., 2023). However, they have not validated this approach for permutations, and we revealed that second-order Taylor approximation fails to accurately estimate the barrier value in the case of permutations. Zhou et al. (2023) introduced the concept of layerwise linear feature connectivity (LLFC) and showed that LLFC implies LMC. Additionally, Zhou et al. (2023) demonstrated that weak additivity for ReLU activation and the commutativity property are satisfied, then both LLFC and LMC are satisfied. However, we show that the 1,2 distance between the models after permutation is not close enough to satisfy the commutativity property. This motivated to investigate the relationship between LMC and WM.
Model merging. Relevant topics of LMC include model merging and federated learning. McMahan et al. (2017) and Konečný al. (2016) introduced the concept of federated learning, where model is trained on divided datasets. Wang et al. (2020) proposed a federated learning method by permuting each component unit and then averaging the weights of the models. Singh & Jaggi (2020) proposed a method for merging models by performing alignments of model weights using optimal transport, which is similar to the method proposed by Ainsworth et al. (2023). Although their method is designed for model fusion and its performance is inferior to that of Ainsworth et al.'s 's method, it can be considered an LMC-based method because it uses hard alignments for the same architecture. Wortsman et al. (2022) proposed a method to improve test accuracy without increasing inference cost, unlike ensemble methods, by averaging the weights of models fine-tuned with different hyperparameters.
Low-rank bias. Empirically, some previous studies (Tukan et al., 2020; Arora et al., 2018; Alvarez & Salzmann, 2017; Yu et al., 2017; Denton et al., 2014) on the compression of trained DNN models have pointed out that even when the weights are replaced with low-rank matrices, the accuracy does not decrease significantly. This suggests that SGD has an implicit bias toward reducing the model weights to low-rank. Yunis et al. (2024) explored the reduction of rank during SGD-based training and the orientation of top singular vectors via SVD on the weights. While this is relevant to our
14

---

Published as a conference paper at ICLR 2025
paper, Yunis et al. (2024) do not examine the effects of permutations in detail. Moreover, it does not discuss the interplay between hidden layer inputs and top singular vectors under permutations. Galanti et al. (2022) and Timor et al. (2023) further discuss the low-rank effect on the weights of trained models introduced by weight decay and a small initialization scale. In particular, Galanti et al. (2022) state that the weights become more low-rank by strengthening the weight decay and increasing the learning rate, and this is expected to help establish LMC via WM (which we experimentally confirm in Appendix H.5).
B CALCULATION OF R WITH THRESHOLD y
This section describes how to calculate the R value with threshold y > 0. Given two models, O. and Ob, let W
(a) and W(O) be the weights of the f-th layers of the models O, and O,, respectively. (a)
Let Σ; u P. (U )T and Σ, u $2,5 (V be the SVDs of these weights. Also, let s!a) and
(b) (b)
((6) be the maximum singular values in all the layers of the models 0, and Os, respectively, The value with the threshold y is calculated as follows:
(0 0.) Eess 11( 2 2s(0))
(s 2 7s ( (u))) (u (
E, min (a) mn (0))
(a) and ny (B) the numbers of singular values greater than 75(4) and 70(0), respectively, and where ni is an indicator function that returns one if the given logical expression is true and zero if is false.
Finally, will briefly explain that RR (Oa, 0.)l (TlyT and w Σ. u [[( (b)
holds. We first define the weight matrices (b)
by w- = E, From the definition, we have:
(a] ugs [[$ 2 78(4))) ((02))
2 7s (((
ur ( (10) Tw
Σ
(0) 70 (o (8) 16
2 78 2 I (u (1 (1 (9)
Note that the i-th singular values of these weights Wifa) and w (b) can be regarded as [( 2 7
and [|s!0 >31"),, respectively. Therefore, von Neumann's 's trace inequality (von Neumann, 1962)
yields that:
tr (Wlaa) TW,(1)) =
[|s Enis (T (b) 2 - = min(n! (a) m [.))
(10)
From Equations (9) and (10), we have:
Enl 2 (a)) (8 z 75 (( (0)) (( ((0 S min n
By summing both sides for l, we get RR, (0,, 0,)l < 1.
C SIMPLE EXAMPLE OF THEOREM 4.2
In Section 4.4, we explained that even when the 2 distance between the weights of two models is large, their outputs be close depending the input distribution. Here, use simple example for a more detailed analysis.
Consider the weights of two models, W(a) and (6), given by:
0.398
0.003
0.210
0.255
-0.319
-0.559
(a) =
1.059
0.303
0.521
w
1.031
-0.155
0.484
0.609
0.785
-0.235
0.742
-0.114
-0.604
The SVDs of these matrices represented by:
(4)) (4) U(aa g(a)
(T)
0.260
-0.127
0.957
(1.317
0
0
0.974
-0.077
0.213
~
0.850
-0.501
0.165
0.959
0.043
-0.859
-0.510
0.458
0.856
0.238
0
0.277
-0.222
-0.506
0.833
15

---

Published as a conference paper at ICLR 2025
and
( = (())) (V(6))
0.266
0.587
0.767
11.317
0
0.974
-0.077
0.213
0.850
00.237
0.470
0
0.958
0
0.188
-0.249
-0.950
0.458
0.774
-0.437
0.277
-0.126
-09965
0.228
respectively.
In this case, the distance between these weights is | [W W || = 1.236. On the other hand, if the input vector given by z (0.974 -0077 0.213), where an arbitrary (but not too large) real scalar value, then (W - o (b) =||l z holds.
D EXPERIMENTAL SETUP
This section describes the experimental setup for training neural networks to obtain SGD solutions. We apply Sinkhorn's algorithm for permutation based WM and STE. Thus, we also provide detailed information on the experimental setup for Sinkhorn's algorithm. Four datasets were used in this study: MNIST (Lecun et al., 1998), Fashion-MNISS (FMNIST) (Xiao et al., 2017), CIFAR1O (Krizhevsky & Hinton, 2009), and ImageNet (Deng et al., 2009).
All experiments were conducted processors, eight NVIDIA A30 GPUs, and 512 GB of memory. The Py ToTcc 2. 2.1.0*, PyTorch Lightning 2.1.0 and torchvision 0.16.0% libraries were used for model training and evaluation.
Linux workstation with AMD EPYC 7543 32-Core
D.I MODEL TRAINING
MLP on MNIST and FMNIST. Following the settings in (Ainsworth et al., 2023), we trained a Multi-Layer Perceptron (MIP) with three hidden layers, each comprising 512 units. The hidden layers use the ReLU function as their activation function. For the MNIST and FMNIST datasets, optimized using the Adam algorithm with a learning rate of 10-3, The batch size and maximum number of epochs were set to 512 and 100, respectively.
VGG11 and ResNet20 on CIFAR1O. We utilized the VGG16 and ResNer20 architectures of (Ainsworth et al., 2023). To accomplish Linear Mode Connectivity (LMC), we increased the widths of VGGII and ResNet20 by factors of 4 and 16, respectively. As described in (Jordan et al., 2023), we used the training dataset repair the BatchNorm layers these models during model merging. Optimization was conducted using Adam with learning rate of 1 x 10-3 The batch size and maximum number of epochs were set to 512 and 100, respectively. The following data augmentations were performed during training: random 32 32 pixel crops, and random horizontal flips.
ResNet50 on ImageNet. ResNet50 models were trained using a training script published on GitHub" by the FFCV library (Leclerc et al., 2023). The "mn50.40 40..epocs.yyame yaml"" file in the repository was used for the training setup. the file, we changed use blurpool to "0". As described in (Jordan et al., 2023), we repaired the BatchNorm layers in these models during model merging by using the training dataset. Since ImageNet a large dataset, we used 50,000 randomly selected images from the training set repair the batch normalization layers.
D.2 PERMUTATION SEARCH
For permutation search in WM and STE, we employed the method based on Sinkhorn's algorithm as proposed by Guerrero-Pein et al. (2023). We utilized the implementation provided by the authors in their GitHub repository DistL2Loss and MidLoss were used as loss functions for permutation
html
16

---

Published as a conference paper at ICLR 2025
searches corresponding to WM and STE, respectively, Optimization was performed using Adam with a learning rate of 1 for MLP, VGGII, and ResNet20 and 10 for ResNet50, setting the maximum number of epochs to 10 for DistL2Loss and five for MidLoss. For MidLoss, the batch size set to 512; for Dist Loss, there was no batch size because the dataset was not used. 100 iterations of parameter updates were performed per epoch for Distl .osss.
For activation matching (AM), the permutation search is divided for each layer, so the optimal solution can be obtained efficiently. We implemented AM-based permutation search following the GitHub repository released by Ainsworth et al. (2023)'. In this paper, the linear.swue -sueaassssssssss innment function of Scipy (Virtanen et al., 2020) was used for the permutation search in AM.
E DISCUSSION ON COMMUTATIVITY PROPERTY
Zhou et al. (2023) show that LMC is satisfied if weak additivity for ReLU activations and commutativity hold. Given two models, 0, and ts. commutativity between them is satisfied
if for all layers & E [L]. w/() ---1 + w((0) T--1
(b) = holds, where be rewritten as - = 0. Therefore, Zhou al. (2023) in Section 5.2 justify the WM-based permutation search method because WM aims to minimize Equation (I), which corresponds to the first factor in the equation. However, as shown in our paper, WM only slightly
(b) w(()) 2/-1 + w
(b)
L is the number of layers of the models 10 The commutativity property
Vl E [L]: (W/()) (=
reduces the distance between the two models, contradicting their claim.
Appendix B.5 of (Zhou et al., 2023) explains in a different way why the commutativity property is satisfied in WM. Specifically, they consider a stronger form of the commutativity property: (11)
(a) VE E [L]; w f-1 w zz A w (b) 200) = W!! 2/-1-
(u)
To ensure this stronger form, we need to find P. and P such that:
(W( P.W. P. )= = 0 (P.W (b) P.T, w Pt-12 1 =0
(12)
They argue that the commutativity property easily holds because it is easy to find permutation matrices Pt and Pt- 1 that satisfy Equation (12) due to the small actual dimension of the hidden layer inputs
*t-1 and 1 (i.e., these vectors are biased in a particular direction).
(a)
However, several points need to be addressed regarding this explanation. First, if Equation (11) holds, then LMC is satisfied without any assumptions, such as the commutativity property or weak additivity of ReLU activations. Since this equation must hold for all layers, it must also hold for the input layer (b) where 2,-1 = z = x. In that case, the outputs of the input layers are equivalent between the two models. The same holds for subsequent layers, so the outputs of the two models are the same in all hidden layers. Therefore, the outputs of the hidden layers of the merged model must also be identical to those of the pre-merged models in all layers. Thus, LMC obviously holds. In other words, if the reason of the establishment of LMC is that Equation (11) holds, then the essential reason for the establishment LMC is that WM makes the outputs of the hidden layers the two models close, suggesting that our argument in this paper is more fundamental in establishing LMC.
(6) Second, the small actual dimension of the hidden layer inputs 2(x)) and = does not necessarily mean that Equation (12) is easier to satisfy by finding permutations that minimize Equation (1). As shown in Section& 4.3, WM preferentially aligns the directions of singular vectors with large singular
(a) values, while other singular vectors are difficult to align. If the hidden layer inputs z and = are
(())
not oriented in the same directions as the right singular vectors with large singular values, then WM
will not help satisfy Equation (12). Zhou et al. (2023) did not mention this second point. On the other hand, we analyzed this point in Section 4.4.
F CONVOLUTIONAL LAYERS
This section discusses theorem similar to Theorem 4.1 for convolutional neural networks (CNNs).
Phtpps: / thub. com/ samuela a/gi t- t-rrebbain
"Strictyy speaking, given a data distribution D, the commutativity property is satisfied if the equation almost surely holds for D. This definition is cquivalent Zhou al. (2023), although differs slightly.
17

---

Published as a conference paper at ICLR 2025
F.1 NOTATION
We introduce the notation used in the following sections. Each element of tensor is specified by simple italic variable with subscripts. For example, for third-order tensor X, its i, k-th component denoted by Xi.. We also use Python-like slice notation. For example, X1,; denotes the first row
of the matrix X. For complex matrix X, let X*= X the complex conjugate of X.
be its unitary transpose, where X denotes
F.2 MATRIX REPRESENTATION CONVOLUTIONAL LAYER
This subsection introduces the matrix representation of convolutional layer. Let X R=xп and Y E RR xлx* be the input and the output of the ê-th convolutional layer, respectively. Here, m denotes the number of input and output channels and n denotes the size of the height and width of the input. For simplicity, we assume that the numbers of output channels and input channels are identical, as well as the sizes of the height and width the input, although our analysis applicable even when they are not. Let K E I x be the kernel of the f-th layer. Then, for the c, r, i-th element of the output Y given by
Yen.
Σ
Xdrtpiit KeaiitgKppqccc K.q..dd
There exists a matrix M such that vec(Y) Salakbutdinov, 2014), where
Mvec (X) holds (Sedghi et al., 2019; Jain, 1989;
B1.1
B1
B
B2,1
B2
B2m
M
(13)
Bm,1
Bm,
Bmm
Here, for all c, E [m], Bed is doubly circulant matrix defined by
cire K Kinaad) ciro K (na)))))
cire( (22,ea)) circ (1,iad)
circ Kninod) circ (
Badd
circ( (2,eed)
circ cire(Ks..e.))
cire(Ki..ce
where circ is function generate circulant matrix from given vector. For example, given vector
011
02
ag
(a1, 02 .s., the circulant matrix generated by is given by circ(a)
0g
a1
022
03
F.3 SINGULAR VALUE DECOMPOSITION AND WRIGHT MATCHING OF CONVOLUTIONAL. LAYERS
Since Equation (13) represents the matrix form of the convolutional layer, we reach conclusion similar to Theorem 4.1 by performing a singular value decomposition (SVD) on it. However, this matrix is very large, with a size of mn² mn? making direct SVD impractical. Therefore, decompose it into a more SVD-friendly form using a Fourier transform. Using the imaginary unit as 77 = 1-1, and setting w e 2m//n a dieeesinniio Fourier transform matrix is defined by F., F,, (v( ",, A matrix for the two-dimensional Fourier transform can be defined as ( Fn. Here, @ denotes the Kronecker product. By using this two-dimensional Fourier transform matrix Q. the matrix M can be decomposed as follows:
M (I X Q) L & Q),
Usually, the alphahet letters or are used for the imaginary unit, but since they are used as indices here,
77-
18

---

Published as a conference paper at ICLR 2025
where In denotes the identity matrix of size m. We then have
D1.1
D1.2
D
D2,1
D2.2
D2.
L
Dm.1
Dm.2
Dmm
Here, for all e, d [m]. De,d QB, is complex diagonal matrix (Sedghi et al., 2019). Let G..,w be matrix formed by extracting the w-th diagonal element of each diagonal matrix De.d and arranging them (i.e., Ge,d,a ) Then, the following theorem holds:
w.w)).
Theorem F.1 (SVD of convolutional layer). Lel 8w,6 Ww and Uw U.. be the i-th singular value, left singular vector, and right singular vector of G.,,.. respectively. Then, the matrix M representing the convolutional layer can be decomposed as follows:
M ( & Q¹ Q'ew)) Sm, (w Q'ew)"
where ew represents the orthonormal basis in Euclidean space R==", and Sw. Uw Q' ewo and Vw,i @ Q'ew ew are the singular value, left singular vector, and right singular vector of M, respectively,
The proof is shown in Appendix G.3. From this theorem, the following theorem be proved:
Theorem F.2. Let M (a) and M (b) be the matrix representations of convolutional layers of two CNNs.
From Theorem F.1, their SVDs are given by (o) = Ew,; (u!!)) e*ew)sw (v @ Q'ew(" and ( = (u
(b) Q' )s (v & Q' e' respectively. Then, the WM between ( and (b) is equivalent to finding permutation matrices Pe and Pe-1 such that
arg max R
Σ
(v (m (Peu (b) (a) ( w
Pt.
where Rz is the real part of z. for complex number
The proof is shown in Appendix G.4. Similar the case of MI.P (Theorem 4.1), Theorem F.2 indicates that WM has the effect of aligning the directions of the corresponding singular vectors in convolutional layers.
G PROOFS
G.1 PROOF OF THEOREM 3.1
Proof. From the assumption and Taylor theorem centered at 0g, we have
1
C(0.) = L(O.) + (0. - -.) + (0. 0.)) H. ( 0a) + +(10 - -0 12)
2
= C(O.) + (1- x)8pV(((0.) + 7 (1- x)232 nt H.u + 0(33).
Similarly, using Taylor theorem centered at T T(,)), we get
1
((O.) ((=(O.)) + (0. [( + 2 ( ((,)) Ho (0,)) + 0 T(.))1s)
= =[(((O))) - A + 2 x332 μT Haju + 0(33).
Combining these equations, the barrier can be obtained
((.,:((0.)) = max (L(0.) - xC - (1 x)((T(()))
A
max (A(C(()) - ((0.)) (1 -)) ((T(()))))
max (BA(1
- VL(T(())))
+ ト 3eA11 2)p ((1 A)H, AHa 14) + +0(8))
19

---

Published as a conference paper at ICLR 2025
G.2 PROOF OF THEOREM 4.1
Proof. Consider the L norm of the f-th layer ( P.W, P.-1112. norm can be rewritten using trace, we have
Using the fact that the 1
[ (a)
P. (b) P. 1| = tr (w (a)
P.W. (b) P.Ii) P.w Pi_i)T)
= ur W (a) )T)
+ lr W W (W(
2tr P.W. (b) (W
(14)
We focus on the last term because only it depends on the permutation matrices. The SVDs of the weights W and will are denoted by W U.a) 5ta) (V(()) E, u s, (()) T and
w/ = U S ( = Ej Ui F.j (1t))) T, respectively. Thus, the last term of Equation (14)
(b) (b)
can be rewritten as
2tr ( (M) P. (w((")) - Σ
(a) (6) (a)
( (PP
(PPul
(b)
Ee $65
fd
Therefore, Equation (1) equals
arg min 10a π r0,))l² arg max Σ >=(Pe) E,3
8ef 22. (T
(0) 16) 16
which completes the proof.
G.3 PROOF OF THEOREM F.1
Proof. Note that the matrix can be decomposed L Ew G.., w @ () by using the tensor
G, where e is the orthonormal basis in Eucrlidean space R"" Thus, the SVD of L is given by
L
2 ( T)
(um Vire)) 6 (e
w
Σ
Σ
Sw,i & (w) & ew)
Thus, the SVD of M is also given by
M
Σ
Sw, 1 II & (uw, (w) (Im) )Q)
S P (w 2 @wee (w 8 &e.T).
w
which completes the proof.
G.4 PROOF OF THEOREM F.2
Before proving Theorem F.2, we first prove the following lemma:
Lemma G.I. Let K and K' be kernels of convolutional layers. We have ||M M n²|k K'I where M and M' are the matrix representations of K and K', respectively.
Proof. From the definition of M and M', we have
B1,1
B
B
B' B
f1
B
1.2
B
B2
B
B
B
B
M
M'
Bmn
Bm
B Bmn
B m
B' m..
B'
20

---

Published as a conference paper at ICLR 2025
where for any c, [m]. Bed and B'dd denote the doubly circulant matrices obtained from the
kernels K and K'. Thus, ||M - Ec,d |Badd all n Eed Ei (iie(((
ciro( Ki Ki = n² Ee,d Ei,(K (Kij.ad K Ki = n² K K'12 holds.
Proof of Theorem F.2. In convolutional layers, permutation matrices permute the input and output channels of the kernel. Therefore, the permutation matrices P: and P:-1 corresponding to the input and output are m X m matrices. By using these matrices, the permutation of the matrix representation of the convolutional layer of the model 0, is denoted by (P& @ I. M (Pt-1 @ mm) )m)T. Lemma G. indicates that finding the permutation matrices that minimize the 2 distance between the two kernels is equivalent to minimizing
(M) - (P& @ Im M (h) (Pt-1 @ [ Therefore, we have |M (P& & In) M (b) (Pr- @ 1
(Pr-1 8 1mm) (Pr Im) (m = ir -1((M( (Pr 2 Im M (P- @ Im)) -
-
(P:-1
tr ( (M + M (M((((M) (M () - tr (M (Pr M (h) (P-- 2 Im)
(0) (Pe-1 @ I..) (M tr (Pr M
(15)
In Equation (15), the permutation matrices Pt and Pt-1 are only related to the last two terms. Therefore, we focus only on them. By using some properties of trace, we have
-t (M ( tr (M (PP (m (Pr-1 @ mm) ))) tr ((M( ( Im M & Im)
Im M (P-1
(P-- & Imm")² - tr (Pe I (P:- & Im) Im) (MM(())))
tr (M ( (Pr- (M [m) )mm )))
(M ((Pr M
(Pt- &
= 1 (M I (Pr 1m) )
Here, from Theorem F.I.
M Σ (u (a)
e (6) e.)
ws
e'e
[ e Q)
(16)
and o M (Pt-1 8 1m) = Σ
(P X Im (u wa 2 Q' Qwew)(
( 8 (eeee))(PPPP @ Im))
(b) wa (Peu (b)
(P-10
(P
(b) QQ
(17)
hold, where we let clo) Σg w. ( and c(O) = E; (00 (P (PL10))) From
Equation (16) and Equation (17), we have
-2 (M)()
(Pa & I. ( (b)
(P-- 8 Imm) {m)))²
E( (0) @ Q' e @ieeeeer (e)
= tr
Σ (Cia)
T
=-2R tr
Σ Cifa) (C
(b)
Q
21

---

Published as a conference paper at ICLR 2025
Using the fact that if w # w', then e O, and otherwise, enjew 1, get
(a) ( ( M (PP km))
-2R tr
= - tr (0) [teccic
@
-2R Σ tr C (C tr
w
22 Σ
tr
(Cl (C
w
( PP ))
Σ -2
tr
(b) (Iu (6)
(Pt- (PP
-2
From the above, the minimization of [| - (Pr M')
(P--1 Im] |T|
is equivalent to the
maximization of R Ew Eis S (u!!))) (Pau)) ((v!))) ((P-i)))))
(b)
(b)
G.5 PROOF OF |R(O,, (0,))) $1
This subsection proves the following theorem: Theorem G.2. Let 0a and O, be the parameters of two MLPs with L-layers. For any permutation π = (Pi) we have
(uig( (1) ((Pru (Pp-1)/)
RiO,. (0.))
$1.
(18)
Eeng ng
The equality holds if. for all i, ue. = Paul
(b) and v 1
(a) Proof. If. for all l,i, Eg Pru and vg we prove that Equation (18) holds. Using the property of trace, we have
1b)
(a)
= P then the equality obviously holds. Thus,
(b)
Σ T(P () ((P-o)))) E,5
(P.u T ( (a) (PP-10()))
Σ tr
Σ
Σ (PaR)) (PP
Let W = (u (o (1)))) and w7Th) E, (Pru (P--o,
( Obviously, they are matri-
ces with all singular values of and thus by using von Neuman's trace inequality (von Neumann, 1962), we have
Elr((w (w (T)
Σ nec
Therefore, the triangle inequality yields that
(4)
Σ ir ( (W
=
ir (withh) (W
which completes the proof.
22

---

Published as a conference paper at ICLR 2025
[|03 -1(00]]
800
Test loss
Test acc.
0.75
750
2
0.50
15
12
18
15
epoch
epoch
epoch
(a) Results of VGGII
- -000)]
700
Test loss
1.5
Test acc.
0.8
1.0
650
0.6
epoch
epoch
epoch
(b) Results of ResNet20
Figure L distance between two models, test loss, and accuracy of merged models when optimizing permutations using Sinkhorn's 's algorithm for WM. Five permutation search trials were conducted with independently trained models (i.e., ten independently trained models were prepared to form five model pairs, and WM was performed for each pair). These results are plotted in different colors.
G.6 PROOF OF THEBRREM 4.2
Proof. Because is Lipschitz continuous with the constant C, we have
E (llo (a) 2) oW
(b) =)1 S CBElWW w!th =1| S C /VEllw(">= w x|13
(19)
where we use Jensen's 's inequality since the squre root function is concave. Focusing on the difference between the outputs in the square root, we get
| (0) (
(b) 2|1 (W T
w!!) (a) x+2 (WW(O) (b)
w!!h (b) 2z (ww((a) TW
have = =TW (o) Ww()) Es 2 (a) uie s; Vi and w( Σ, (b) (b) (b)
From the SVDs of weights w(() Σ.
(a) (a) (a)
ug ii
2) =T(w!!) Tw (. and
=(Ww(a) w (b) z Eis 8i
(b) (u (o(o)) T((0))) z. Therefore, Equation (19) can
be rewritten as
Ella(W!!") 2) (w(()))) =)1l
sc
Σ (o ( T>) Σ (5 (((
Σ
( u E(0 [=()()) Tzz
H ADDITIONAL EXPERIMENTAL RESULTS
H.1 LEARNING CURVE WM
In this subsection, Figure 6 shows the learning curves for VGGII and ResNet20 when WM is performed using Sinkhorn's 's algorithm. The figure shows that the distance between the two models decreases the training progresses, and the performance of the merged model also improves. In this paper, for both VGG1| and ResNet20, we used permutations at the 10th epoch, when the loss the merged model is stably small.
23

---

Published as a conference paper at ICLR 2025
DC1
1C2
3
FC
Singular value
10-
10-
Coan
Dlny
Capy 1
5
5
0
0
500
0
0
500
0
500
0
Coex
2000
2838320000
Cley
Cesy
20000
100-
150
Iodex
103
(a) MLP, MNIST.
Stapular
54-
50
FC1
FC2
PC3
PC
2000053093
10000
20000
Singllar value
10-
10
30-
Index
VGGI1, CIFARIO.
(b) ML.,, FMNIST.
Conv
Conv 2
Conv
Conv
Conv
Conv
15
25-
20
20
20
20
10
0
0
0
2500
250000
250000
250000
250000
0
250000
Conv 7
Conv 8
Conv 9
Conv 10
Conv 11
Conv 12
50
25
50
50
50
50
Singular value
250000
250000
100000
0
250000
100000
0
100000
Conv 13
Cony 14
Conv 15
Conv
Conv
Conv 18
100-
100
50-
50-
50
0
0
0
0
100000
100000
100000
50000
100000
50000
Conv 19
Conv 20
Conv 21
FC 1
100
100
2.5
2.0
0-
0
1.5
50000
50000
50000
5
Index
(d) ResNet20, CIFARIO.
Figure 7: Distributions of the singular values of each layer.
MLP MNISTT
MLP FMNIST
VGGLL CIFARIO
ResNet20, CIFAR10
2.00
Singular value
Singuiar valun 6.
Siguuar valee
2.5
1.75
3.50
1.50
3.25
Singular
2.0
1:28
1.00
1.00
0.75
1.5
0
Index
Index
Index
Index
Figure 8: Adjusted distributions of the singular values of the output layer.
24

---

Published as a conference paper at ICLR 2025
H.2 DISTRIBUTION OF SINGULAR VALUES
Figure 7 shows the distributions of the singular values of all the layers. The figure demonstrates that, in all layers except for the output layer, the singular values very similar across all models. Meanwhile, in the output layer, there is variability in the singular values. However, this variability (a) (0) does not affect the accuracy of the merged model. Let WL E, LL (I(")),, and W
(b) (b) (0)
1 1. (v)..) represent the output layer weights of the two trained models. The figure shows that the difference between the singular values of the two models is approximately constant multiple. In other words, there exists a constant a such that (a) ~ ass for all i. confirm calculated and the weight of the output layer is adjusted, demonstrating that correcting the output layer by constant factor can address the differences the distribution. If the singular vectors of the two weights are equal (i.e., L
this, Figure shows the distribution of singular values when the constant
(a) u for all i), then W! = aW holds (indeed, as mentioned in Section 4.3, the permutation matrix aligns the directions of the singular vectors). Therefore, the weight of the output layer of the merged model the ratio A E [0. 1] is given by XW(2) + +1 (1 = WW = (1 (1-A)a W L = (A (A+ (1 W Thus, we can consider that the weight and the activation function of the merged model are given by w( and softmax function with inverse temperature of 1/(A+ (1 - X)a), respectively. Since the inverse temperature does not affect the accuracy value, the difference the singular values of the output layer would not matter in satisfying LMC, at least in terms of accuracy.
(() = 2 and L.
(a)),
H.3 INNER PRODUCTS BETWEEN RIGHT SINGULAR VECTORS OF HIDDEN LAYERS AND THEIR INPUT
Figure shows the average absolute values of inner products between the right singular vectors and the input in each layer of models trained by SGD. The figure demonstrates that, except for the input and output layers, the singular vectors with larger singular values generally have larger inner products with the inputs. Note that the results of the input layers do not affect the permutation search based on WM because their right singular vectors are not changed by the permutations.
25

---

Published as a conference paper at ICLR 2025
Expre 1
bayer
Layer3
Exyre
ed Layer
Ted Luger
Layec
10
50-
509
50000
259
0
2000
50000
20000
560
500
Lay 4
Luysr
Layer
Indexx
25000
2000
5000
(a) ML.,, MNIST.
500000
10000
20000
Laywr 7
Laywr
Layer
5000
5000
100
100
5:50
5000
5000
Indixx
Index
(b) MIP, FMNIST.
(c) VGGII, CIFARIO.
Layer
Layer 2
Layer
Layer
Layer 5
Layer 6
16
1e7
1e6
1e7
16
1e7
1
5
2.5
0
0.0
0.0
0
2500
250000
250000
0
250000
250000
250000
Layer 7
Layer 8
Layer
Layer 10
Layer 11 1e6
Layer 12
1e7
1e5
1e5
5
2.5
0.0
0
250000
250000
0
100000
0
250000
100000
100000
EE E/v'2/
Layer
Layer
Layer
Layer
Layer
Layer
1e6
1e6
1e6
1e5
1e6
1e5
2
2.5
0
0
0.0
100000
100000
100000
50000
100000
50000
Layer 19
Layer 20
Layer 21
1e5
1e6
1e6
Layer
1
100-
50
0
50000
50000
50000
Index
(d) ResNet20, CIFARIO.
Figure 9: Average absolute values of inner products between the right singular vectors and the input of each layer. The horizontal axis represents the index of the left singular vector, while the vertical axis shows the mean square the inner product. The left side of the horizontal axis corresponds to singular vectors with large singular values.
26

---

Published as a conference paper at ICLR 2025
Singular value ratio
VGG11
Singolar valun ratio
ResNet20
1/8
x
1:4
x
0.5
0.5
0.0
0.0
0.5 Ratio
1.0
0.5
Ratio
Figure 10: Distribution of all singular values normalized by the largest one in the model as the model width multiplier changes. The vertical axis represents the singular values divided by the maximum singular value of each model, and the horizontal axis represents the ratio among all singular values (e.g., the point at 0.5 on the horizontal axis represents the singular value in the middle of all values sorted in descending order).
H.4 RELATIONSHIP WITH MODEL WIDTH
Previous studies have demonstrated that the width of the model architecture affects the ease of achieving LMC. In this subsection, we explain this phenomenon based on the following three facts: as the model width increases, (i) the proportion dominant singular values decreases, (ii) the right singular vectors corresponding to these dominant singular values will have large inner product values with the inputs of the hidden layers, and (iii) the WM preferentially aligns the directions of singular vectors corresponding to these dominant singular values.
(i) Dependency of model width on singular values. As we mentioned, the proportion of relatively large singular values in all singular values decreases the model width increases. To verify this, Figure 10 shows the distribution of the singular values of all layers of VGG11 1 and ResNer20 trained on CIFAR1O. Figure 10 shows the results of different model widths (i.e., dimensionality). As can be seen, the proportion of relatively large singular values decreases the model width increases. Thus, the proportion of singular vectors that need to be aligned in the model decreases as the width increases.
(ii) Dependency of model width on inner products of right singular vectors. We also investigated the effect of model width on the inner products between the hidden layer inputs and the right singular vectors. Figures 11 and 12 show the values of these inner products for each layer as model width changes. Figures 11 and show the distributions of inner products for VGG11 and ResNet20 models trained on CIFAR10, respectively. These figures demonstrate that as model width increases, the inner products between the right singular vectors with large singular values and the inputs also increase.
(iii) Singular- alignment. We conducted experiment to examine how well the directions of singular vectors are aligned as model width increases when applying permutations found by WM. The results are shown in Figure 13. The figures display the evaluation of R(O,, ग (0.)) for the trained models O2 and Og by searching for permutations π. For comparison, the case where no permutations are applied (i.c., æ is identity map) is also shown. Additionally, a threshold y was introduced to assess the alignment of singular vectors with large singular values.
First, focusing on the results in Figure 13(a) with 7 O, we observe that the value of R decreases even when the width increases and WM is used. Conversely, Figure 13(b) shows that the directions of singular vectors with particularly large singular values are aligned by permutation as model width increases. This suggests that even with WM, it is difficult to perfectly align the directions of singular vectors between the two models. However, increasing the width decreases the fraction of singular vectors with large singular values, thus making it casier for WM to align the directions of these dominant singular vectors.
As shown in Figure 10, when the model is sufficiently wide, the proportion of large singular values is very small compared to the total number of singular values. Furthermore, Figures 11 and 12 demonstrate that the right singular vectors associated with these relatively large singular values have a large inner product with the hidden layer input. This means that the number of singular vectors that WM needs to align to achieve LMC is reduced when the model is wide enough, discussed in
27

---

Published as a conference paper at ICLR 2025
Lest Lagec1
10 Layer
Lager
Legt Lagee
cG Layec
Exper
20000
20000
2000
Lagec
Layer 5
Layer
Layec
Layec
Layer
10000
EIVE
1000
250
0-
2000
500
1000
2500
1000
2000
Layec 7
Layer B
Loger
Layec 7
Layer
Layer 9
100
20
100
200
200
5
500
500
(a) Width multiplier 0.125.
(b) Width multiplier is 0.25.
LeG Taywr!
1e5 Laper
Laywe
LeG Lagne
16 Layer 2
Layer
20000
25000
0
2000
5000
2500
2000
10000
5000 Layer 6
Tagwr
Layer 5
Loyee
Layen
Layer 5
500-
500
5000
2000
2500
10000
2500
5000
Layer?
Layec
Lagne
Layer
150
150
100
50
50
1000
1000
2000
Index
Index
(c) Width multiplier is O.S.
(d) Width multiplier
Layse1
LeG Layer
Eayer
LeG Lagee
16 Layer 2
Exyer
50000
2000
20000
10000
2000
500000
20000
Laywe
2000
5000
10000
2000
Blva
000
10000
50000
10000
20000
Laywe
Layer
Layer
Layee
Layer
Lxywr
20.
5000
2000
1000
160
0
2500
Index
Index
(e) Width multiplier is 2.
(I) Width multiplier is 4.
Figure 11: Average absolute values of inner products between the right singular vectors and the input of each layer of VGG11 trained on CIFAR10.
28

---

Published as a conference paper at ICLR 2025
Lager
Layer 2
Layer 3
Layer 1
Layec
Layer 2
Laysr
Layer
1GG
1eG
1c6
1e6
1oc
1o6
1ec
2.5-
2500
10000
10nno
10000
2500
25000
25000
2,0000
Layer 5
Layer 6
Layer
Lagera
Layer
Layer
Layer
Layer
Inn
Iw7
H 10000
10000
10000
10000
25000
25000
25000
2 25000
1e5
Layur 9
le?
Layar 10
1e5
Layar 1.
Lugur 12
Te
Layar 9
Laywe 10
1ed
Layse 11
1es
Layer 12
2.5-
50000
0.0-
Elvzz
5000
5000
23000
10000
10000
Layer 13
Layer
Loper 13
Layer 15
Iwi5
Layer 14
Laver 16
Layer 14
1m0
Layer
50000
5000-
50000-
20000
Masialm 10000
5000
5000
5000
10000
1000n
5000
Layer
17
Lager 188
Layer
Loger
Layec 18
Layse 19
Layae 20
10000
2500-
20000
50000
2500-
50000
5000
2500
2500
2500
10000
5000
5000
5000
25:00
Layer 21
Layrer 22
2500
Layer
50
Layes 22
2500
6
5000
Index
Index
(a) Width multiplier is 1.
(b) Width multiplier is 2.
Loger
Layer 2
Layer
Lager
Layer
Layer
Layer
Layer
tell
2.5
Iefl
1d
lef
le7
2.5
1
25 oo
50000
5oúc0
S000e
2000
100000
100000
10000n
Laywrr
Layer
Laywr?
Layer'3
Layer
Layer
106
1:6
1:7
1e6
167
2.5
2.5
50000
50000
50000
50000
100000
100000
100000
100000
Layer
Layer 10
Lager 11
Layer
Layer's
Lager 10
Layer 11
Layer 12
1w
Iw?
twb
166
1w7
1w5
2.5
1-
2.5
0.0
0,0
0.0
0.0
EVVap
250000
25000
25000
100000
50000
50000
Laywr
Lager 14
Layer
Laywr
Layer
Layer
Luywr
IeG
Laywr
2-
25000
25000-
25000
25000
25000
10000
50000
50000
5000
20000
Layer
Layer 18
Layer
Layer
Layer
1ob
Lo5
Layer 19
Leb
165
16
Layer
105
50000
5000
0-
0
25000
10000
10000
10000
5,0000
20000
20000
200000
Layar 21
5000
Layar
75
Layer
1:5
100-
Layer
50-
10000
5
20000
Index
Index
(c) Width multiplier is 4.
(d) Width multiplier is 8.
29

---

Published as a conference paper at ICLR 2025
Liyer
Layer
Layec 3
Layer
16
L67
1eG
167
2000.00
0.0
200000
0.0
200000
Layer
Layer
Layer 7
Eayer
1e0
1+2
5-
2.5
200000
2000
200000
200000
Lay
Layer
L67
1e6
165
2.5
100000
200000
100000
100000
Layer 133
Layer 14
Layer 15
Tager 16
Jnf
1w3
100000
1000000
100000
50000
Luyse 17
Layer
Layar
Laywr 20
2.5-
100000
50000
50000
50000
Lagwrr 21
LG
Layer 22
100
0-
Index
(e) Width multiplier is 16.
Figure 12: Average absolute values of inner products between the right singular vectors and the input of each layer of ResNet20 trained on CIFAR10.
VGG1 1
ResNet20
VGG11
ResNet20
0.5
0.3
0.5
0.3
W/ WIM
wi WM
Net WM
w/ WM
Value I
0.2
w/b WM
Value of I
0.2
wib WM
0.2
wib WIM
0.2
Wio WM
0.1
Width multiplier
Width muitippler
Width maltiplier
Width maitiplier
(a) Evaluation results of ((00.; for all the singu-(b) Evaluation results of Ii(O,, x0.)) for the singular vectors with 0.3.
lar vectors.
Figure 13: Relation between the model width and the difficulty in aligning the directions of singular vectors.
Section 4.4. Indeed, Figure 13(b) suggests that WM preferentially aligns these significant singular vectors. Therefore, increasing the width is expected to make LMC more feasible.
30

---

Published as a conference paper at ICLR 2025
Test loss of merged model
Test loss original model
Proportion of large singular values
0.5
weight decay - 1e-05
0.4
waight decay 36-05
2.0
2.0
Proportion
0.3
weight docay 0.0001
weight decay 0.0003
1.5
1.5
weight decay 0.001
0.2
weight decay
1.0
1.0
0.1
0.5
0.5
0.0
10
10
10.7
10
10
10
10
10"
20
Learning
Learning
Learning
(a) Evaluation results of ResNet20 16).
Test loss of merged model
Test loss of original model
Proportion of large singular values
0.20
weight decay 1e-25
2.0
Proportion
0.15
weight decey 0.0001
waight docay 36
1,055
1.5
3:5-
0.10
Weight decsy 0.001
Weigght decay 0.0003
waight
1.0
1.0
0.05
0.5
0.5
0.00
30
30-3
10-2
10
10-3
10-a
10
10-3
10-2
Leaminn rate
Learning rate
Loaming rate
(b) Evaluation results of VGGII (x2).
Figure 14: Experimental results of model merging with WM under different learning rates and weight decay strength.
H.5 DEPENDENCY OF WEIGHT DECAY AND LEARNING RATE
Qu & Horvath (2024) have observed that strengthening weight decay and increasing the learning rate make it easier for LMC to be established through WM. In this section, we will explain this observation from the perspective of singular values of weights.
In Section 4, we stated that the reason why WM can establish LMC is that the permutation found by WM aligns singular vectors with large singular values between two models. In other words, the smaller the proportion of large singular values in the weights of each layer of the models, the more likely LMC to be established by WM. In fact, some previous studies (Galanti et al., 2022; Timor et al, 2023) have shown that increasing the weight decay and learning rate during model training can reduce the ranks of the weights in the trained model. Therefore, it is highly likely that the results observed by Qu & Horvath (2024) were caused by the reduction in the ranks of the weights. In the following, we will experimentally confirm this prediction.
Figure 14 shows the experimental results of ResNet20 and VGGI I models. During model training, the learning rate was varied from 0.0001, 0.0003, 0.03 and the weight decay strength was varied from 0.00001, 0.00003, -.. 0.003. For each condition, six models were trained, and model merging was performed three times by creating three pairs from them. The conditions for model training were the same as in Appendix D. except for the learning rate and weight decay. For VGGI1, when the model width is quadrupled, the ratio of large singular values becomes very small regardless of the learning rate or weight decay, making difficult to understand the relationship between the loss of the merged model and large singular values. Thus, the model width was doubled for VGG11 models. In addition, the permutation search method used in WM was based on the method of Ainsworth et al. (2023).
Figure 14 shows the test losses of the merged model and the original model, as well as the ratio of the number of large singular values to the width of the original model. Figure 14 displays the averaged results over three runs of model merging. The ratio the figure was calculated as follows. Let s1,1, ... st be the singular values of the f-th layer, where ni represents the number of singular values, and $1, is the largest singular value. Each singular value is divided by the largest singular value, and those whose ratio is 0.3 more are counted (i.e., $t,5/st, 2 0.3 for E [me]). Next, for all layers, we calculate the sum of these numbers and divide it by the sum of n, for all layers. This procedure can be written as Σe, /8../80.1 0.3)/ Ee nig, where is an indicator function. In
31

---

Published as a conference paper at ICLR 2025
Table 4: Model Merging results with AM and the estimated barrier value using Taylor approximation when À = 1/2
Datasct
Network
Test acc.
Barrerr (A = 1/2)
Taylor approx.
|00 0.
18, x(6)
Reduction rate [%)
CIFARIO
2esNer20
VGGI1
88. 786
190 0.192
1 0.186
0.077 1: 0.044
0.189 + 0.031
7,431 00.67
2.491
1: 0.2266
799. 4: 16, 116
Y10.762
16. 261
671.373
742.300 4: 18. 525526
13.816
5.538
7.161 4: 0.572
0.226
FMINIST
MEP
85.350
0.221
-02230 = 0.029
0.313
0.173
121.853
5.830
108, 90S
5.305
10.678 = 0.343
MNIST
MPP
98.274 0.084
0.0220 + 0.004
0.064
0.035
81.231 + 5.580
68.722 + 5.197
15.428 + 1.037
RIB, m(8.))))
0.4
AM
w/o AM
0.2
0.0
V-0.3
y-0
V-D.3 MLP FMNIST
y-0
y-0.3
y-
y-
y- CIEAR10
MLP, MNIST
VGG11, CIRARIO
ResNe120,
Figure 15: Evaluation results of R(O,, 0s) with and without AM.
other words, this ratio becomes smaller when the number of singular values that are relatively large compared to the maximum singular value is small compared to the width of the model.
From Figure 14, we can see that the ratio of large singular values decreases as both the weight decay and the learning rate increase. This has already been noted in previous research (Galanti et al., 2022; Timor et al., 2023). Additionally, Figure 14 shows that the test loss of the merged model decreases when both the test loss of the original model and the ratio of large singular values are small. This can be explained by our analysis in Section 4. As we described, WM facilitates LMC by aligning the singular vectors between the two models and making the functions of the middle layers of the merged model and the original model more similar. In other words, this suggests that it is challenging for the merged model to outperform the original model. Furthermore, the smaller the ratio of large singular values, the easier it align the singular vectors using WM, the functions of the merged model and the original model become more closely aligned. From this, we conclude that the higher the performance of the original model and the smaller the ratio of large singular values, the better the performance of the merged model. This is consistent with the results shown in Figure 14.
H.6 ACTIVATION MATCHING
Table shows the results of model merging using AM under the same conditions as Section The table indicates that the loss barrier is sufficiently small when AM is applied. Interestingly, AM reduces the distance between the two models to the same extent as WM.
Figure 15 shows the value of R between the two models O, and Og. The figure clearly shows that the value of R is larger using AM when y = 0.3. This indicates that AM aligns the directions of singular vectors with large singular values for the two models, similar to the result of WM (Figure 2). To further examine the relationship of singular vectors between the models before and after merging, Figure 16 shows the values of R between these models. This result also demonstrates similar trend to the WM results shown in Figure 3, suggesting that the reasons for the establishment the LMC almost the same for both WM and A.M.
Rist8o
8.8
5.6
56
64
ML. MNIST
NLR SINNDST
VGSE1, CIPARED
Recheeso
CITARIO
MLR MISITT
NLR PAENIST
VGG11, CIPARE
Reckne20 CLIALL
(a) Evaluation results with the threshold ? = 0.
(b) Evaluation results with the threshold y = 0.3.
Figure 16: Evaluation results of R value between the merged model and the pre-merged models (i.c., R(O.. (0, + π (0)))/2) and R(O,, ... (0,, + T (0.)))2)) when AM is used. The blue and red bars represent the evaluation results of R(O., (0. x (0a)) and R(O,, (0, ((+)))/2) respectively.
32

---

Published as a conference paper at ICLR 2025
Table 5: Evaluation results barrier between each pair models
burnie
Accuracy barner
10, 2
10,
+0,02
10,
+o100//7
10. 1.,0111
WM
CIPARIO
REaNe120
VOGTT
0004 + 0.168
0.020
0.007 4.0.19
0,088
+0.11
0.14 0:141
00888
2300
7.50 4.958
10:12 6117
1:23
1:103
4,337
MNIST
MEP
00027
0.005
0.044
0.005
-0,01
6.008
0.04
0.475 0.060
CI
VOGIT
0.001
0,088
0.022
2.172
01989
4.86
5.70
32:013
15,005
3.783
0.016
0333 0.055
2,007
16.77
MINIST
MIPP
00057 0.011
DO00 I 0.006
0X01 0014
0.263
0.176
0.368 00.16
2312 0.40
MIP MNIST
MLP FMNIST
ResNet20 CIFAR10
VGG11 CIFAR1O
60
6.00
80
89.66
600
89.66
83.45
40
400
91
400
-7.03
77.24
2.00
200
58.62
20
52.41
46.21
20
40
60
50
100
200
400
600
250
750
40.00
(a) Results of WM.
MLP MNIST
MLP FMNIST
ResNet20 CIFAR10
VGG11 CIFAR10
8036
600
95.86
89.66
600
83.45
60
400
71 71.03
40
64.83
200
58.62
20
60
52.41
0-ai
46.21
0
50
100
200
800
500
750
40.00
(b) Results of STE.
Figure 17: Accuracy landscape around 0.. To (0,) and (e.).). The star in the lower left represents O.. and the squares in the lower right and upper represent To (0,). and (O.), respectively.
H.7 STE AND WM
In this subsection, additional experimental results for Section 6.3 are shown Table 5 for the barrier values between each pair of models. The table shows the model-merging results with À 1/2, and the mean and standard deviation of three model merges. the table, a negative value for the barrier indicates improvement in performance due the merging. In addition, Figure 17 shows
the accuracy landscape around 0., (O,),, and (O.).. From Table 5 and Figure 17, we can see that the barrier between 7h(00, (O,) and (0.) is also smaller for WM than for STE.
H.8 DEPENDENCY OF R ON THRESHOLD 7
Figure 18 shows the value of R when the threshold y is varied. Figure 18(a) displays the value between model 0₂ and the permuted model r(O,), along with the R value before permutation for comparison. Figure 18(b) illustrates the R values between the merged model (f,, + T (0.))/ /2 and the original models 0,, and 0,, also including the R value without permutation for comparison.
In Figure 18(a), the value between the original models is nearly zero regardless of the ? value. increases. This indicates that WM preferentially aligns the directions of the larger singular vectors between models 9,, and 0,. As shown in Figure 18(b), this effect helps align the singular vectors with larger singular values between the merged and original models. Aligning these singular vectors more closely makes LMC more feasible because the outputs between the two models are closer, discussed in Section 4.4.
However, when permutation is applied, the It value increases
33

---

Published as a conference paper at ICLR 2025
MLP MINSST
MUP FMNIST
VOG11 CHIAROO
ResNet20 CIEARIO
0.50
0.50
0.50
0.50
R10,- 0,)
0.25
0.25
0.25
0.25
RIO., ml0.))
0.00
0.00
0.00
0.00
0.0
0.2
0.4
0.0
0.2
0.4
0.0
0.2
0.4
0.0
0.2
0.4
(a) R value between the models O, and O,.
1.0
MLP MIISST
1.0
MLF IMNIST
1.0
VIG11 CHPARIO
1.0
ResNetzo CHIMR1I
10, a.12)
0.8
0.8
22
Rte. 18, 4.32)
0.6
0.6
0.6
0.6
10, (66.122
0.4
0.4
0.4
0.4
Rlatesl. 18, m16.2/2)
0.0
0.4
0.0
0.4
0.2
Y
y
(b) R value between the merged model and the original model.
Figure 18: R values when the threshold y is changed.
Table 6: Results of model merging of ResNet50 models trained on ImageNet dataset.
(a) Test loss and top-1 accuracy of each model.
(b) L distance between 9g and 0,.
0. w/
0g w/o
0.
0g
L" dist. w/ WM
L dist. w/o WM
Test loss
5.207 + 0.073
6.897 + 0.001
1.491 = 0.011
1.493 = 0.007
Test acc.
40.239 + 2.088
0.179 0.020
75.741
2.088
75.836 0.107
126.823 0.533
174.247 1 0.577
H.9 LMC ON RESNET5O TRAINED ON IMAGENET
In the paper, most of the analysis was performed on relatively small datasets such as MNIST and CIFAR1Ó. In this subsection, we train ResNet50 models on larger dataset, ImageNet, and analyze the results of model merging based on WM.
Experimental results of model merging. Table 6 presents the results of merging ResNet50 models trained on the ImageNet dataset. Table 6(a) shows the test loss and top-1 accuracy of the models before and after merging (i.c., the pre-merged models 02 and 0,, and the merged model O.). Table 6(b) shows the L distance between models O2 and O, before and after applying permutations. These tables report the mean and standard deviation for five independent model merges. According to Table 6(a), the test loss and accuracy of the merged model are clearly improved by using WM. However, they are still worse than those of the pre-merged models 9, and 0,, indicating that the LMC cannot be considered satisfied. Table 6(b) demonstrates that using WM decreases the L distance between models O,, and On. This decrease in L distance is larger than that observed for VGG11 and ResNet20 in Table 1, suggesting that the singular vectors of models O, and 6, are better aligned by permutations. Figure 19 presents the results of evaluating R for each model pair. When y 0.3, Figure 19 shows that R(Oa, 0p) rises to about 0.5 with WM, and the value of R increases to approximately 0.9 between the pre- and post-merged models (i.e., R(O.. 9z) and R(O,, f.)). Thus, even with ResNet50, the singular vectors with large singular values are aligned between the pre- and post-merged models by using WM.
To investigate why LMC does not hold even though the singular vectors with large singular values are aligned by using WM, we examine the distributions of singular values at each layer and the inner products between the right singular vectors and the inputs. Figure 20 shows the distribution of singular values for each layer, and Figure 21 shows the distribution of the average absolute values of the inner products between the right singular vectors and the inputs for each layer. Each figure is plotted in a different color for the 10 trained models. Figure 20 demonstrates that the distributions of singular values are nearly identical across all models. Thus, the difference in singular values between models is not reason for preventing LMC. In Figure 21, focusing on the distribution of the inner product between the right singular vectors and the inputs, we observe that the right singular vectors with large singular values do not necessarily have large inner product with the input. As discussed in Section 4.4, WM can only align singular vectors with large singular values, so this discrepancy can be reason preventing the establishment of LMC. As shown in Figures 11 and 12, the wider
34

---

Published as a conference paper at ICLR 2025
R(O,. 9p)
RIO,, 0.)
R(0,. 0z)
1.0
1.0
value
w/ WM
value
w/ WM
R value
W/ WM
0.5
w/o WM
0.8
w/o WM
0.8
w/ WM
0.6
0.6
0.0
0.3
0.3
0.3
y-
Figure 19: Evaluation results of R values between each pair of ResNet50 models trained ImageNet dataset.
the model, the larger the inner product of the input and the right singular vectors with large singular values in the hidden layers. Therefore, it is considered necessary to increase the width of the model to establish LMC with ResNet50.
35

---

Published as a conference paper at ICLR 2025
Cony 1
Cony 2
Cony 3
Cony 4
Conv 5
Conv 6
2.5
2.5
10
0
0
0.0
0.0
0
100000
200000
200000
200000
200000
200000
Conv 7
Conv 8
Conv 9
Conv 10
Conv 11
Conv 12
2.5
1.0
0.0
200000
200000
200000
200000
200000
0 250000
Conv 13
Conv 14
Conv 15
Conv 16
Conv 17
Conv 18
5
2.5
1-
0.0
0
0
250000
100000
500000
100000
100000
100000
Conv 19
Conv 20
Conv 21
Conv 22
Conv 23
Conv 24
1.5
1.0
2.5
0.5
0.0
0
0.0
100000
100000
0
100000
100000
100000
100000
Singular value
Conv 25
Conv 26
Conv 27
Conv 28
Conv 29
Conv 30
2.5
5-
5
0.0
0
0
200000
200000
50000
250000
50000
50000
Conv 31
Conv 32
Conv 33
Conv 34
Conv 35
Conv 36
2
1.5
5
2.5
0.5
0
0.0
0
50000
50000
50000
50000
50000
50000
Conv 37
Conv 38
Conv 39
Conv 40
Conv 41
Conv 42
2.5
1.5
1.5
1.0
2
1.0
1-
0.5
0-
0
50000
50000
50000
50000
50000
50000
Conv 43
Conv 44
Conv 45
Conv 46
Conv 47
Conv 48
2
2
0.0
50000
100000
0
1 100000
25000
0
200000
0
25000
Conv 49
Conv 50
Conv 51
Conv 52
Conv 53
FC 1
10
5-
2.5
0
0
0.0
25000
25000
25000
25000
25000
1000
Singular vector index
Figure 20: Distributions of the singular values of each layer of ResNet50 models trained ImageNet dataset.
36

---

Published as a conference paper at ICLR 2025
Layer
Layer 2
Layer 3
Layer 4
Layer 5
Layer
5000
2500
2500
250
500
50
100000
200000
200000
200000
200000
200000
Layer
Layer
Layer
Layer 10
Layer 11
Layer
200
100
100
200000
200000
200000
200000
200000
250000
Layer 13
Layer 14
Layer 15
Layer 16
Layer 17
Layer 18
50
25
10
25
10
11s
4M
0
250000
100000
500000
100000
100000
100000
Layer 19
Layer 20
Layer 21
Layer 22
Layer 23
Layer 24
10
2
Je'N
0.0
100000
100000
100000
100000
100000
100000
Layer 25
Layer 26
Layer 27
Layer 28
Layer 29
Layer 30
1E|v22|2
10-
2.5
2.5
0
0.0
0.0
200000
200000
50000
250000
50000
50000
Layer 31
Layer 32
Layer 33
Layer 34
Layer 35
Layer 36
0.25
0.2
0.5
0.25
0.00
0.0
0.00
0.0
0.0
50000
50000
50000
50000
50000
50000
Layer 37
Layer 38
Layer 39
Layer 40
Layer 41
Layer 42
0.5
0.1
0.5
0.1
0.1
0.1
0.0
0.0
0.0
0.0
0.0
0.0
50000
50000
50000
50000
50000
50000
Layer 43
Layer 44
Layer 45
Layer 46
Layer 47
Layer 48
0.1
0.5
0.05
0.1
0.5
0.0
0.0
0.00
0.0
50000
100000
100000
25000
200000
25000
Layer 54
Layer 49
Layer 50
Layer 51
Layer 52
Layer 53
1e-5
0.2
0.01
0.005
0.005
0.025
0.00
0.000
0.0
0.000
0,000
25000
25000
25000
25000
1000
Singular vector index
Figure 21: Average absolute values of inner products between the right singular vectors and the input of each layer in ResNet50 trained on ImageNet dataset (x (x10%).
37


Reviews:
[{"review_id": "87jIyqBA0L", "reviewer": "Reviewer_Q5i3", "summary": "This paper analyses the the weight matching algorithm through singular value decomposition. \n\nPaper starts with the observation that algorithms like weight matching do not significantly reduce the  l2 distance. Followed by this observation, authors empirically verify that weight matching algorithm does not bring the models close enough such that the error barrier can be calculated by a second order Taylor approximation. \n\nAuthors then made the observation that minimizing the l2 distance wrt permutations is equivalent to maximizing the inner product wrt permutations. Using this relationship authors show that weight matching indeed matches the singular values of the layer-wise weight matrices. \n\nAuthors empirically validate that weight-matching and activation matching align the singular vectors however STE does not match the STE. \n\nAuthors empirically show that aligning via SVD can lead to better alignment for 3 models.", "questions": "- What explains that permutations found by activation matching are much different from weight matching? Can we use SVD to analyze the qualitative / quantitative differences between the two methods that ostensibly minimize a similar objectives, i.e. matching the singular values.\n- In [2], authors also analyzed linear connectivity of more than 3 models for checking strong linear connectivity where it is found that as width of the model increases, weight matching allows strong linear connectivity. Are there any insights we can gain from SVD when it studying strong linear connectivity.\n\n[2] Sharma, E., Kwok D., Denton T., Roy DM, Rolnick D., and  Dziugaite G.K.: Simultaneous linear connectivity of neural networks modulo permutation. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (2024).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Analysis of the weight matching via SVD of layerwise weight matrices is interesting.\n- I find the observation that weight alignment leads to linear connectivity of more that 2 models also interesting", "weaknesses": "- I would suggest authors to keep the vectorized notation instead of summations to make the paper an easier read.\n- Observation that minimizing the l2 distance is equivalent to  maximizing the inner product is pretty well know. [1] also leverages this observation to implement the weight matching algorithm. I found those contributions incremental.\n\n[1] Ainsworth, S., Hayase, J., Srinivasa, S.: Git re-basin: merging models modulo permutation symmetries. In: The Eleventh International Conference on Learning Representations (2023)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wPBHRLgVet", "reviewer": "Reviewer_hUsi", "summary": "This work presents a novel analysis of weight matching (WM) for model merging, with comparisons to AM and STE. The authors demonstrate that permutations found by WM may not reduce the L2 distance between 2 models, albeit LMC is satisfied. Instead they show that the singular vectors are aligned. They also demonstrate that STE merging does not align singular vectors.", "questions": "- Connecting it back to the idea in the intro about SGD and the loss landscape, do the authors have any thoughts on how their findings relate to the efficacy of SGD?\n- How well does this analysis scale beyond 2 or 3 models? \"Therefore, WM is likely to be more advantageous, especially for merging three or more models\" (line 522), is there additional proof of this?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This work covers a very important topic likely to have a broad impact in understanding learning and combining in deep neural networks. \n- Paper is very well written; notation is clear and consistent, the problem is motivated, contributions are clear. \n- The Taylor approximation to estimate the barrier for merging models provides a clear intuition for their approach, and demonstrate that WM permutations may not result in the best merge. Results in Table 1 provide good evidence for this stance (though perhaps too many unnecessary significant figures are given and appear arbitrary as a result). \n- Results overall support the paper claims and demonstrate the different merging means for WM, STE, and AM.", "weaknesses": "- Table 1 (and others) could be improved through highlighting sections to support the conclusion, e.g., where some solutions violate the nearness assumption and the Taylor approx. is not good. Many values presented are overlapping with over values and it is not immediately clear which are significant different vs not. \n- Figures are small and difficult to read. For the distribution plots (Fig. 1, 4), perhaps a logscale on the y axis would allow the reader to better evaluate the claims. \n- \"the barrier between πb(θb) and πc(θc) is smaller with WM than with STE. This means that there is a significant difference between the principles of permutations obtained by WM and STE.\" (line 519) is difficult to evaluate by eye in Figure 6 and not particularly convincing. \n- The statement \"suggesting that the reason AM achieves LMC is likely the same as for WM\" (line 458) seems overly confident given the evidence presented. Are there other explanations for the similar results?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cfp0uTjtRD", "reviewer": "Reviewer_PdBc", "summary": "This paper analyses the linear mode connectivity (LMC) using weight matching (WM), activation matching (AM), and straight-through estimator (STE), with a primary focus on WM. The authors propose that, even though WM does not significantly decrease the $L_2$ distance of the weights between two independently trained models, it still establishes LMC by aligning the singular vectors of each layer associated with the largest singular values. Empirically, the paper demonstrates that two models with a large $L_2$ weight distance after WM can nonetheless produce similar outputs, and empirical results support that singular vectors with largest singular values across the layers are aligned with WM process. Furthermore, it has been demonstrated that AM process exhibits a similar effect, while STE behaves differently. The theoretical analysis leverages SVD on layer weights, and numerical experiments using VGG, ResNet, and MLP on MNIST, CIFAR10, and ImageNet support the claims.", "questions": "* For the sentence in line 139 \"Entezari et al. (2022) conjectured that for SGD solutions ...\": is this still a conjecture, or has a more solid theoretical foundation been established?\n\n* In line 122, why is $W_{\\ell + 1}' = W_{\\ell + 1} P^{T}$ while $W_\\ell' = P W_\\ell$? I thought it should be $W_{\\ell + 1}' = P W_{l+1} $", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is clearly written and easy to follow. \n\n* I have checked the proofs of the theorems presented in the main paper, and they appear correct and clearly articulated.\n\n* The observation that $L_2$ distance reduction via WM is not the sole cause of LMC is interesting. This observation is supported with numerical experiments.\n\n* The theoretical analysis is clearly explained when the underlying assumptions hold (though I have questions about the assumptions' validity; see the Weaknesses section).\n\n* The appendix provides a good amount of detail for reproducing experiments, although the code itself is not shared.", "weaknesses": "* In line 70, the sentence \"... this intuition is inaccurate ...\" raises a concern. I think the intuition presented above this sentence is correct, i.e., if $\\theta_a \\approx \\pi (\\theta_b)$ then it is reasonable to think that the interpolated model will produce similar outputs. I believe the main narrative of the paper appears to be that even when $\\theta_a \\not\\approx \\pi (\\theta_b)$, where $\\pi$ is found by WM, the interpolated model can produce approximately equal outputs. \n\n* In line 211, the phrase \"... algorithm similar to the backpropagation method to efficiently compute $\\mu^T H$ ...\" would benefit from further clarification in the appendix. I couldn’t locate a section elaborating on this point. What specifically do the authors mean by \"similar algorithm to backprop\"?\n\n* Regarding the sentence in line 261 \"... we can find permutation matrices such that the left and right singular vectors of the two models match ...\": it seems unlikely that all singular vectors can be matched purely through permutation in all scenarios.\n\n* In line 286 \"distributions of the singular values of the weights ... almost equal\": This seems like a strong claim to me. The paper presents results with common datasets like MNIST and CIFAR10 using MLP, VGG, and ResNet. I think the conclusion requires further evidence or a more theoretical foundation. The appendix H.1 discusses this in a more detailed manner, and Figure 7 indicates that output layers' singular value distribution have variability. For this case, the paper suggests that the singular values are approximately a constant multiple across the two models. Even though the output layers' singular value distributions seem correlated, the \"constant multiple\" assumption seems like a strong one to me as there are contradicting examples in Figure 7 (e.g. Figure 7(a) light blue curve and red curve).\n\n* The assumption that the input vector's direction aligns with the singular vectors associated with large singular values also seems like a strong claim. Analysis in Appendix H.8 with ResNet50 on ImageNet shows that this assumption does not always hold.\n\n* The abbreviation \"layer-wise model compression (LMC)\" in line 890 conflicts with the abbreviation for linear mode connectivity, and I believe layer-wise model compression does not need an abbreviation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bhek7UUkHS", "reviewer": "Reviewer_oBtL", "summary": "This paper analyzes the algorithms for enforcing linear mode connectivity (LMC) as discussed in Ainsworth et al. (2023). The central question addressed is the underlying mechanisms that lead to the removal of barriers when applying weight matching (WM), activation matching (AM), or the straight-through estimator (STE). The authors argue that the primary reason WM and AM achieve linear connectivity between models is the alignment of the eigenvectors corresponding to large eigenvalues of the weights. Theoretically, they demonstrate that similar eigenvectors lead to similar outputs at the layer level, and that minimizing the L_2 distance between weights is equivalent to aligning these eigenvectors. Furthermore, the paper suggests that STE operates through a different mechanism to achieve low-barrier perturbations.", "questions": "1 - Both WM and AM are iterative algorithms. How does the L_2 distance evolve with an increasing number of iterations? It is surprising that the distance shrinkage is nearly identical between WM and AM, given that AM corresponds to layer-wise feature matching, which is closely linked to reduced distances between models.\n\n2 - While the experimental results indicate that the Taylor approximation is imprecise for estimating the barrier, could it be that the models are still relatively close? Since L_2 distance is difficult to interpret in high-dimensional spaces, do you think that including the third term of the Taylor decomposition might help, or that a more precise analysis of the barrier (not just at the midpoint) could yield better insights?\n\n3 - How does your analysis account for the necessity of post-matching fine-tuning, which is particularly critical for networks with batch normalization layers (e.g., Jordan et al., Repair: Renormalizing Permuted Activations for Interpolation Repair)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The study offers a valuable perspective on analyzing LMC, a phenomenon that characterizes the properties of loss surfaces and solutions discovered by stochastic gradient descent (SGD) and its analogs. The results are presented clearly and are substantiated both empirically and, in part, theoretically.", "weaknesses": "A key weakness is the lack of significant novelty in the paper's primary insights. Similar observations have been made in previous works, such as [1] and [2]. These papers also explore Hessian-based barrier approximations and spectral analysis to explain LMC. Although this does not undermine the findings of the current paper, these existing works should be cited and discussed.\n\nAnother concern is the transition to per-layer analysis in the discussion of WM. The authors claim that eigenvector alignment results in aligned representations, but this can be challenged. Firstly, the use of the Lipschitz constant in the analysis limits the result to essentially one activation only. When multiple layers are involved, the Lipschitz constant becomes excessively large. Secondly, [3] show that there are often no barriers at the layer level (even when barriers exist at the full network level). Although this could still be explained by eigenvectors, it weakens the connection between similar activations on the level of one layer and the lack of barriers at the full-model level.\n\nThe analysis of STE seems rather straightforward, given that this algorithm is designed solely to minimize loss at a single point (the midpoint on the linear path). Unlike WM or AM, it does not aim to identify fully \"matching\" models. The numerical results in Table 4 are also not fully convincing, as the difference between the barriers found using WM and STE is minimal.\n\nAdditionally, the paper claims that models trained in the same manner exhibit higher barriers at the midpoint between them. However, in the experiments of [3], it is shown that when one model generalizes better than another, the higher barrier shifts. Therefore, the assumptions in Section 3.2 require further clarification.\n\nMinor Comments:\n\n1 - In Section 4.4, please specify that the results in Appendix H3 and H4 are empirical.\n\n2 - I suggest replacing Figure 6 with Table 4, as low-dimensional visualizations provide an imprecise representation of loss surface properties.\n\n[1] Singh et al., Landscaping Linear Mode Connectivity\n\n[2] Yunis et al., Approaching Deep Learning through the Spectral Dynamics of Weights\n\n[3] Adilova et al., Layerwise Linear Mode Connectivity", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)