Analysis of Linear Mode Connectivity via Permutation-Based Weight Matching: With Insights into Other Permutation Search Methods

## Abstract

Abstract Recently, Ainsworth et al. ( 2023 ) showed that using weight matching (WM) to minimize the L 2 subscript 𝐿 2 L_{2} italic_L start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT distance in a permutation search of model parameters effectively identifies permutations that satisfy linear mode connectivity (LMC), where the loss along a linear path between two independently trained models with different seeds remains nearly constant. This paper analyzes LMC using WM, which is useful for understanding stochastic gradient descent’s effectiveness and its application in areas like model merging. We first empirically show that permutations found by WM do not significantly reduce the L 2 subscript 𝐿 2 L_{2} italic_L start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT distance between two models, and the occurrence of LMC is not merely due to distance reduction by WM itself. We then demonstrate that permutations can change the directions of the singular vectors, but not the singular values, of the weight matrices in each layer. This finding shows that permutations found by WM primarily align the directions of singular vectors associated with large singular values across models. This alignment brings the singular vectors with large singular values, which determine the model’s functionality, closer between the original and merged models, allowing the merged model to retain functionality similar to the original models, thereby satisfying LMC. This paper also analyzes activation matching (AM) in terms of singular vectors and finds that the principle of AM is the same as that of WM. Finally, we analyze the difference between WM and the straight-through estimator (STE), a dataset-dependent permutation search method, and show that WM can be more advantageous than STE in achieving LMC among three or more models.

## 1 Introduction

Deep learning has significantly advanced various fields, including image classification, speech recognition, and natural language processing (Vaswani et al., ; van den Oord et al., ; Zhao et al., ). Large-scale neural networks (NNs) are widely used in these applications, and optimizing their parameters poses a massive non-convex optimization problem. Remarkably, stochastic gradient descent (SGD), which is widely used for training NNs, is known to find good solutions despite its simplicity. One hypothesis for this seemingly counterintuitive phenomenon is that the landscape of the loss function may be much simpler than previously thought. Several studies (Garipov et al., ; Draxler et al., ; Freeman & Bruna, ) have found that different neural network solutions can be connected by nonlinear paths with almost no increase in loss. Recently, Entezari et al. () conjectured that 1.1 holds, considering all possible permutation symmetries of NNs:

###### Conjecture 1.1 (Permutation invariance, informal) .

Let $\bm{\theta}_{a}$ and $\bm{\theta}_{b}$ be two SGD solutions (model parameters). Then, with high probability, there exists a permutation $\pi$ such that the barrier (defined in Definition 2.1) between $\bm{\theta}_{a}$ and $\pi(\bm{\theta}_{b})$ is sufficiently small.

Here, the barrier represents the increase in loss observed when linearly interpolating between the weights of the two models. A sufficiently small barrier implies that the model obtained by linear interpolation of the two models performs the same as the original models. If the barrier between two models is sufficiently small, we say that linear mode connectivity (LMC) is satisfied between them (Frankle et al., ). 1.1 suggests that most SGD solutions can be transferred into the same loss basin using permutations. Indeed, some studies (Ainsworth et al., ; Singh & Jaggi, ) have experimentally demonstrated that this conjecture is valid for various datasets and models using weight matching (WM), which identifies permutations that minimize the $L_{2}$ distance between the weights of two models.

Understanding LMC principles based on permutation symmetries is important not only for comprehending how SGD works in deep learning but also for its application in model merging (Singh & Jaggi, ), where two independently trained models are combined. The method of finding permutations using only the $L_{2}$ distance is particularly versatile, dataset-independent, and computationally efficient. In fact, several studies (Singh & Jaggi, ; Wang et al., ; Peña et al., ) have proposed applications of permutation symmetries in model merging, federated learning, and continual learning.

The current theoretical analysis of LMC depends on the feasibility of closely matching NN weights using permutations. Recently, Zhou et al. () proved that if the distance between the weights of two models can be sufficiently reduced through permutation, then LMC holds. Intuitively, given two SGD solutions $\bm{\theta}_{a}$ and $\bm{\theta}_{b}$, if a permutation $\pi$ can be found such that $\bm{\theta}_{a}\approx\pi(\bm{\theta}_{b})$, it is reasonable to expect that the outputs of the interpolated model will approximately equal the outputs of the original models $\bm{\theta}_{a}$ and $\bm{\theta}_{b}$.

However, this paper shows that this intuitive explanation is inaccurate and clarifies a more fundamental reason why LMC holds through permutations found by WM. Our analysis shows that even if LMC holds, the permutations found by WM do not significantly reduce the distance between the two models (at most about a 20% reduction). We then explain this in terms of singular vectors with large singular values. Our analysis not only reveals the principle behind WM but also shows that WM may be more advantageous in merging more than three models compared to other methods such as STE.

The contributions of this paper are threefold:

###### Conjecture 1.1 (Permutation invariance, informal) .

#### 1. Demonstrating that the L 2 subscript 𝐿 2 L_{2} italic_L start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT distance reduced by WM is not the direct cause of LMC.

This paper empirically shows that permutations found by WM do not significantly reduce the $L_{2}$ distance between the two models. Our results indicate that, even when LMC is satisfied, permutations reduce the model weight distance by no more than 20%. Supported by a Taylor approximation, our findings suggest that reducing the $L_{2}$ distance through permutations is not the direct reason for LMC satisfaction.

#### 2. Revealing the reason why WM and activation matching (AM) satisfy LMC.

We provide evidence that WM satisfies LMC by aligning the directions of singular vectors with large singular values in each layer’s weights. This alignment ensures that the singular vectors with large singular values, which determine the model’s functionality, become similar between the merged and original models. Additionally, we show that, from the perspective of the input distribution at each hidden layer, aligning singular vectors with large singular values can efficiently approximate the functionality between two models, even if the $L_{2}$ distance cannot be significantly reduced. As a result, the merged model retains functionality similar to the original models, which facilitates LMC. We also conducted experiments with AM and found that the reason why LMC holds in AM is the same as in WM.

#### 3. Revealing STE is fundamentally different from WM in principle, which leads to a significant difference between them when merging multiple models.

To distinguish WM from other permutation search methods that are independent of $L_{2}$ distance, we examine the straight-through estimator (STE), which focuses on minimizing the barrier itself rather than the $L_{2}$ distance. Our experiments reveal that the permutations found by STE do not align the directions of singular vectors, which is a critical difference compared to WM in achieving LMC. Furthermore, we demonstrate experimentally that this difference significantly impacts the satisfaction of LMC among three or more models.

## 2 Background and Preliminaries

### 2.1 Notation

For any natural number $k\in\mathbb{N}$, let $[k]=\{1,2,\dots,k\}$. Bold uppercase variables represent tensors, including matrices (e.g., $\bm{X}$), and bold lowercase variables (e.g., $\bm{x}$) represent vectors. For any tensor $\bm{X}$, its vectorization is denoted by $\textrm{vec}(\bm{X})$, and $\|\bm{X}\|$ denotes its Frobenius ($L_{2}$) norm.

### 2.2 Permutation Invariance

We consider multilayer perceptrons (MLPs) $f(\bm{x};\bm{\theta})$ with $L$ layers for simplicity while our analyses in this paper can be applied to any model architectures. Here, $\bm{x}\in\mathbb{R}^{d_{\mathrm{in}}}$ is the input to the NN, and $\bm{\theta}\in\mathbb{R}^{d_{\mathrm{param}}}$ represents the model parameters, where $d_{\mathrm{in}}\in\mathbb{N}$ is the dimension of the input, and $d_{\mathrm{param}}\in\mathbb{N}$ is the dimension of the parameters. Let $\bm{z}_{\ell}$ be the output of the $\ell$-th layer (i.e., $\bm{z}_{0}=\bm{x}$, and, for all $\ell\in[L]$, $\bm{z}_{\ell}=\sigma(\bm{W}_{\ell}\bm{z}_{\ell-1}+\bm{b}_{\ell})$). Here, $\sigma$ denotes the activation function, and $\bm{W}_{\ell}$ and $\bm{b}_{\ell}$ represent the weight and bias of the $\ell$-th layer, respectively. Note that in this MLP, we have $\bm{\theta}=\big{\|}_{\ell=1}^{L}\left(\mathrm{vec}(\bm{W}_{\ell})\mathbin{\|}
\bm{b}_{\ell}\right)$, where $\|$ represents the concatenation of vectors.

NNs have permutation symmetries of weight space. Considering an NN with model parameters $\bm{\theta}$, for its $\ell$-th layer, $\bm{z}_{\ell}=\bm{P}^{\top}\bm{P}\bm{z}_{\ell}=\bm{P}^{\top}\sigma(\bm{P}\bm{W
}_{\ell}\bm{z}_{\ell-1}+\bm{P}\bm{b}_{\ell})$
holds, where $\bm{P}$ is a permutation matrix. Note that permutation matrices are orthogonal, so we have $\bm{P}^{\top}=\bm{P}^{-1}$. Therefore, by permuting the input of the $(\ell+1)$-st layer with $\bm{P}^{\top}$, the model parameters can be changed without altering the input-output relationship of the NN. Specifically, the new weights and bias are given by $\bm{W}_{\ell}^{\prime}=\bm{P}\bm{W}_{\ell}$, $\bm{b}_{\ell}^{\prime}=\bm{P}\bm{b}_{\ell}$, $\bm{W}_{\ell+1}^{\prime}=\bm{W}_{\ell+1}\bm{P}^{\top}$. Such permutations can be applied to all layers. We denote the tuple of permutations corresponding to each layer as $\pi=(\bm{P}_{\ell})_{\ell\in[L]}$. Moreover, if a model $\bm{\theta}$ is given, the application of permutation $\pi$ to $\bm{\theta}$ is denoted by $\pi(\bm{\theta})$.

### 2.3 Linear Mode Connectivity (LMC)

Let $\bm{\theta}\in\mathbb{R}^{d_{\mathrm{param}}}$ be a model and $\mathcal{L}(\bm{\theta})$ denote the value of the loss function for the model $\bm{\theta}$. Here, we define the loss barrier between two given models $\bm{\theta}_{a}$ and $\bm{\theta}_{b}$ as follows:

###### Definition 2.1 .

For two given models $\bm{\theta}_{a}$ and $\bm{\theta}_{b}$, their loss barrier is defined as

$$ $\displaystyle B(\bm{\theta}_{a},\bm{\theta}_{b}):=\max_{\lambda\in[0,1]}\big{( }\mathcal{L}(\lambda\bm{\theta}_{a}+(1-\lambda)\bm{\theta}_{b})-(\lambda \mathcal{L}(\bm{\theta}_{a})+(1-\lambda)\mathcal{L}(\bm{\theta}_{b}))\big{)}.$ (1) $$

Intuitively, the barrier represents the increase in loss because of the linear interpolation of the two models. Two models $\bm{\theta}_{a}$ and $\bm{\theta}_{b}$ are said to be linearly mode connected when the loss barrier between them is approximately zero.

###### Definition 2.1 .

### 2.4 Permutation Selection

Entezari et al. () conjectured that for SGD solutions $\bm{\theta}_{a}$ and $\bm{\theta}_{b}$, there exists a permutation $\pi$ such that LMC holds between $\bm{\theta}_{a}$ and $\pi(\bm{\theta}_{b})$ with high probability. Afterward, Ainsworth et al. () proposed WM, straight-through estimator (STE), and activation matching (AM) as methods for finding such permutations. This subsection explains WM, which is the main focus of this paper. AM and STE are discussed in Sections 5 and 6.

In WM, we search for a permutation that minimizes the $L_{2}$ distance between two models(^1^11Although only the weights are considered here, the biases can also be dealt with by concatenating the biases and the weights.):

$$ $\displaystyle\operatorname*{arg\,min}_{\pi}\|\bm{\theta}_{a}-\pi(\bm{\theta}_{ b})\|^{2}=\operatorname*{arg\,min}_{\pi}\sum_{\ell\in[L]}\|\bm{W}_{\ell}^{(a)} -\bm{P}_{\ell}\bm{W}_{\ell}^{(b)}\bm{P}_{\ell-1}^{\top}\|^{2},$ (2) $$

where, without loss of generality, let $\bm{P}_{L}=\bm{I}$ and $\bm{P}_{0}=\bm{I}$, and $\bm{I}$ is an identity matrix.
This minimization problem is known as the sum of the bilinear assignments problem, which is NP-hard (Koopmans & Beckmann, ; Sahni & Gonzalez, ; Ainsworth et al., ). Recently, Peña et al. () proposed solving Equation 2 using Sinkhorn’s algorithm (Adams & Zemel, ) by considering it as an optimal transport problem. We adopt their method because it allows for the optimization of all layers simultaneously, unlike the method by Ainsworth et al. (), and potentially finds better solutions.

## 3 Motivating Observations

Previous studies have suggested that the closeness of two parameters in terms of $L_{2}$ distance is important for satisfying LMC. For example, Zhou et al. () showed that LMC holds if a commutativity property is satisfied. This property holds if, for all layers $\ell$, $\bm{W}^{(a)}_{\ell}-\bm{P}_{\ell}\bm{W}^{(b)}_{\ell}\bm{P}_{\ell-1}^{\top}=\bm
{0}$. Zhou et al. () argued in Section 5.2 that since WM finds the permutation that minimizes Equation 2, WM can be seen as searching for permutations that satisfy the commutativity property. In particular, there is a huge number of permutations because the total number of possible permutations grows exponentially as the number of layers and the width increase, and thus, some of them may sufficiently reduce the distance between the two models. However, this section explains from the perspective of a Taylor approximation that this intuition is not always correct. Our results demonstrate that even when LMC is satisfied, the permutations found by WM do not necessarily bring the models as close as expected. The facts observed from the experiments in this section motivate us to explore other reasons for satisfying LMC in the following sections.

### 3.1 Closeness of Two Models in Terms of Taylor Approximation

This subsection describes the estimation of the barrier value using the Taylor approximation. Let $\bm{\theta}_{a}$ and $\bm{\theta}_{b}$ be two SGD solutions, and $\pi$ be a permutation found by WM to make $\pi(\bm{\theta}_{b})$ close to the model $\bm{\theta}_{a}$. Let $\bm{\theta}_{c}=\lambda\bm{\theta}_{a}+(1-\lambda)\pi(\bm{\theta}_{b})$ be the merged model at a ratio $\lambda\in(0,1)$. If $\bm{\theta}_{a}$ and $\pi(\bm{\theta}_{b})$ are sufficiently close, then their linear interpolation $\bm{\theta}_{c}$ should be close to both models. Therefore, the loss of the parameter $\bm{\theta}_{c}$ should be able to be approximated by the Taylor approximation. In fact, the following theorem holds if $\bm{\theta}_{a}$ and $\pi(\bm{\theta}_{b})$ are sufficiently close:

###### Theorem 3.1 .

The loss function $\mathcal{L}:U\ni\bm{\theta}\mapsto\mathcal{L}(\bm{\theta})\in\mathbb{R}$ is assumed to be a twice differentiable function on an open set $U$ over $\mathbb{R}^{d_{\textrm{param}}}$. Let $\bm{H}_{a}$ and $\bm{H}_{b}$ be the Hessian matrices centered at the models $\bm{\theta}_{a}$ and $\pi(\bm{\theta}_{b})$, respectively. If, for any $\lambda\in(0,1)$, $\lambda\bm{\theta}_{a}+(1-\lambda)\bm{\theta}_{b}\in U$ holds, then we have

$$ $B(\bm{\theta}_{a},\pi(\bm{\theta}_{b}))=\max_{\lambda}\lambda(1-\lambda)\big{[ }\beta\bm{\mu}^{\top}\nabla(\mathcal{L}(\bm{\theta}_{a})-\mathcal{L}(\pi(\bm{ \theta}_{b})))+\frac{1}{2}\beta^{2}\bm{\mu}^{\top}\left((1-\lambda)\bm{H}_{a}+ \lambda\bm{H}_{b}\right)\bm{\mu}\big{]}+O(\beta^{3}),$ (3) $$

where $\nabla$ is the gradient with respect to the parameters, $O$ is the Landau symbol, $\beta$ is the $L_{2}$ distance between $\bm{\theta}_{a}$ and $\pi(\bm{\theta}_{b})$, and $\bm{\mu}$ is the unit vector from $\bm{\theta}_{a}$ to $\pi(\bm{\theta}_{b})$ (i.e., $\bm{\mu}=(\pi(\bm{\theta}_{b})-\bm{\theta}_{a})/\beta$).

We prove this theorem in Section G.1. The theorem states that if $\bm{\theta}_{a}$ and $\pi(\bm{\theta}_{b})$ are sufficiently close, the barrier value can be predicted from the gradients and Hessian matrices around each model.

###### Theorem 3.1 .

### 3.2 Experimental Results

**Table 1: Results of WM and the estimated barrier value using Taylor approximation when $\lambda=1/2$. The table presents the mean and standard deviation from five trials of model merging (i.e., the linear combination of the models $(\bm{\theta}_{a}+\pi(\bm{\theta}_{b}))/2$). The columns labeled “Test acc.”, “Barrier”, and “Taylor approx.” show the test accuracy, the barrier value, and the estimated barrier value using Equation 3 for the merged model at $\lambda=1/2$, respectively. The table also shows the $L_{2}$ distance between the models $\bm{\theta}_{a}$ and $\bm{\theta}_{b}$ before and after applying the permutation, as well as the reduction rate of the $L_{2}$ distance (i.e., $(\|\bm{\theta}_{a}-\bm{\theta}_{b}\|-\|\bm{\theta}_{a}-\pi(\bm{\theta}_{b})\|) /\|\bm{\theta}_{a}-\bm{\theta}_{b}\|$).**
| Dataset | Network | Test acc. | Barrier ($\lambda=1/2$) | Taylor approx. | $\|\bm{\theta}_{a}-\bm{\theta}_{b}\|$ | $\|\bm{\theta}_{a}-\pi(\bm{\theta}_{b})\|$ | Reduction rate [%] |
| --- | --- | --- | --- | --- | --- | --- | --- |
| CIFAR10 | VGG11 | $85.456\pm 3.260$ | $0.035\pm 0.1$ | $2.956\pm 0.35$ | $799.503\pm 16.396$ | $746.465\pm 19.576$ | $6.64\pm 0.808$ |
| ResNet20 | $88.808\pm 0.369$ | $0.167\pm 0.035$ | $7.517\pm 0.573$ | $710.762\pm 16.261$ | $661.055\pm 12.539$ | $6.987\pm 0.472$ |  |
| FMNIST | MLP | $88.330\pm 0.438$ | $-0.183\pm 0.049$ | $0.928\pm 0.175$ | $121.853\pm 5.83$ | $100.041\pm 4.71$ | $17.897\pm 0.348$ |
| MNIST | MLP | $98.382\pm 0.116$ | $-0.033\pm 0.006$ | $0.036\pm 0.03$ | $81.231\pm 5.58$ | $64.751\pm 4.795$ | $20.305\pm 1.225$ |

We conducted experiments to verify whether the Taylor approximation (Equation 3) accurately estimates the barrier between two SGD solutions. Table 1 presents the experimental results of model merging. Details about the datasets, network training procedures, and permutation search methods used in these experiments are described in Appendix D. In the table, we chose $\lambda=1/2$ because it is empirically known that the midpoint between two models results in the highest loss (Ainsworth et al., ; Peña et al., ). We used an algorithm similar to the backpropagation method (Bishop, ; Pearlmutter, ) to efficiently compute $\bm{\mu}^{\top}\bm{H}$, which is required for the evaluation of Equation 3. A negative value in the Barrier column indicates that the loss of the merged model is lower than that of the original (pre-merged) models.

The table shows that, except for the MNIST dataset, there is a large difference between the actual barrier values and those estimated by the Taylor approximation. These differences are particularly large for VGG11 and ResNet20. Additionally, the table indicates that the $L_{2}$ distance changes by only about 6% to 20% from the original distance. This suggests that WM does not bring the models sufficiently close, at least not close enough for a second-order Taylor approximation to hold.

## 4 Analysis of WM

The previous section demonstrates that the establishment of LMC by WM is not due to the reduction in $L_{2}$ distance itself, but rather because WM helps find permutations that result in a smaller barrier between the two models. To better understand why WM reduces the barrier, we first analyze WM by performing SVD on the weights of each layer of the model in Section 4.1. Then, in Section 4.2, we show that the singular value distribution of each layer are almost identical across independently trained models, and that the primary differences between the models are due to variations in their singular vectors. In Section 4.3, we demonstrate that WM preferentially aligns the directions of singular vectors with large singular values between the weights of the two models. Finally, in Section 4.4, we explain that aligning singular vectors with large singular values makes LMC more achievable because these singular vectors predominantly influence the outputs of the hidden layers of the models.

### 4.1 Analysis Based on SVD

The basic idea in analyzing WM is to perform SVD on the weight in each layer. Although using SVD for analysis might seem overly simplistic given that WM reduces the $L_{2}$ distance, this approach provides important insights that are explored in subsequent sections. In WM, permutation matrices are searched to minimize Equation 2. We denote the SVDs of $\bm{W}_{\ell}^{(a)}$ and $\bm{W}_{\ell}^{(b)}$ by $\bm{W}_{\ell}^{(a)}=\bm{U}_{\ell}^{(a)}\bm{S}_{\ell}^{(a)}(\bm{V}_{\ell}^{(a)}
)^{\top}=\sum_{i}\bm{u}_{\ell,i}^{(a)}s_{\ell,i}^{(a)}(\bm{v}_{\ell,i}^{(a)})^
{\top}$ and $\bm{W}_{\ell}^{(b)}=\bm{U}_{\ell}^{(b)}\bm{S}_{\ell}^{(b)}(\bm{V}_{\ell}^{(b)}
)^{\top}=\sum_{j}\bm{u}_{\ell,j}^{(b)}s_{\ell,j}^{(b)}(\bm{v}_{\ell,j}^{(b)})^
{\top}$, respectively. Here, we assume that the singular values are ordered in descending order (i.e., for all $\ell\in[L]$, $s_{\ell,1}^{(a)}\geq s_{\ell,2}^{(a)}\geq\dots\geq s_{\ell,n}^{(a)}$ and $s_{\ell,1}^{(b)}\geq s_{\ell,2}^{(b)}\geq\dots\geq s_{\ell,n}^{(b)}$, where $n$ is the number of singular values). Then, we have

$$ $\displaystyle\operatorname*{arg\,min}_{\pi}\|\bm{\theta}_{a}-\pi(\bm{\theta}_{ b})\|^{2}=\operatorname*{arg\,min}_{\pi}\sum_{\ell\in[L]}\Big{\|}\sum_{i}\bm{u }_{\ell,i}^{(a)}s_{\ell,i}^{(a)}(\bm{v}_{\ell,i}^{(a)})^{\top}-\sum_{j}\bm{P}_ {\ell}\bm{u}_{\ell,j}^{(b)}s_{\ell,j}^{(b)}(\bm{P}_{\ell-1}\bm{v}_{\ell,j}^{(b )})^{\top}\Big{\|}^{2}.$ (4) $$

Equation 4 shows that the permutation matrices $\bm{P}_{\ell}$ and $\bm{P}_{\ell-1}$ are multiplied by the left and right singular vectors of the model $\bm{\theta}_{b}$, respectively. If the singular values of both models are almost equal (i.e., for all $\ell$ and $i$, $s_{\ell,i}^{(a)}=s_{\ell,i}^{(b)}$) and we can find permutation matrices such that the left and right singular vectors of the two models match, then the two models $\bm{\theta}_{a}$ and $\bm{\theta}_{b}$ will be exactly the same and LMC clearly holds. Therefore, in the following, we will discuss the differences in (1) singular values and (2) singular vectors of independently trained models.

Figure: Figure 1: Distribution of the singular values in the second layer. The singular values of ten independently trained models (i.e., trained with different seeds) are plotted in different colors. The distribution of the singular values for all layers is shown in Section H.1.
Refer to caption: x1.png

### 4.2 Differences Between Singular Values of Two Models

First, we investigate the differences between the singular values of two independently trained models. To this end, ten models are trained independently under identical conditions except for the seed, and their singular values are compared. Figure 1 plots the singular values in the second layer of independently trained models in descending order. The evaluation results for all the layers are shown in Figure 7. As can be seen in the figures, in the hidden layers, the singular values are very close across all models.
Therefore, the differences in singular values between the models are not a major issue in achieving LMC.

### 4.3 Singular-Vector Alignment

In the previous subsection, we confirmed that the distributions of the singular values of the weights of the two independently trained models were almost equal. In this subsection, we will show that the permutations found by WM only align the large singular vectors of the two models. We will also show experimentally that WM cannot align all singular vectors between the two models, and therefore cannot reduce the $L_{2}$ distance to zero.

First, we introduce the following theorem:

###### Theorem 4.1 .

Given the trained $L$-layer MLPs $\bm{\theta}_{a}$ and $\bm{\theta}_{b}$, Equation 4 is equivalent to

$$ $\displaystyle\operatorname*{arg\,min}_{\pi}\|\bm{\theta}_{a}-\pi(\bm{\theta}_{ b})\|^{2}=\operatorname*{arg\,max}_{\pi=(\bm{P}_{\ell})_{\ell}}\sum_{\ell,i,j} s_{\ell,i}^{(a)}s_{\ell,j}^{(b)}(\bm{u}_{\ell,i}^{(a)})^{\top}(\bm{P}_{\ell} \bm{u}_{\ell,j}^{(b)})(\bm{v}_{\ell,i}^{(a)})^{\top}(\bm{P}_{\ell-1}\bm{v}_{ \ell,j}^{(b)}).$ (5) $$

The proof of this theorem is shown in Section G.2. Focusing on the term for each layer $\sum_{i,j}s_{\ell,i}^{(a)}s_{\ell,j}^{(b)}(\bm{u}_{\ell,i}^{(a)})^{\top}(\bm{P
}_{\ell}\bm{u}_{\ell,j}^{(b)})(\bm{v}_{\ell,i}^{(a)})^{\top}(\bm{P}_{\ell-1}
\bm{v}_{\ell,j}^{(b)})$ in Equation 5, $(\bm{u}_{\ell,i}^{(a)})^{\top}(\bm{P}_{\ell}\bm{u}_{\ell,j}^{(b)})$ is the inner product between the left singular vector $\bm{u}_{\ell,i}^{(a)}$ of the model $\bm{\theta}_{a}$ and the left singular vector $\bm{u}_{\ell,j}^{(b)}$ of the model $\bm{\theta}_{b}$, applied with the permutation matrix $\bm{P}_{\ell}$. The permutation matrix is orthogonal, so it only permutes the elements without changing the norms of the left singular vectors. Therefore, this inner product is maximized when the directions of the two left singular vectors are aligned by the permutation matrix. The same applies to the right singular vectors. Thus, Equation 5 can be interpreted as finding permutation matrices that align the directions of the singular vectors for all layers between two models, especially those associated with large singular values. Like MLPs, Appendix F shows a similar analysis holds for convolutional layers.

Then, to empirically evaluate how well the singular vectors are aligned, we calculate

$$ $\displaystyle R(\bm{\theta}_{a},\pi(\bm{\theta}_{b}))=\frac{\sum_{\ell,i,j}( \bm{u}_{\ell,i}^{(a)})^{\top}\bm{P}_{\ell}\bm{u}_{\ell,j}^{(b)}(\bm{v}_{\ell,i }^{(a)})^{\top}\bm{P}_{\ell-1}\bm{v}_{\ell,j}^{(b)})}{\sum_{\ell}n_{\ell}},$ (6) $$

where $n_{\ell}$ is the number of singular values in the $\ell$-th layer. Note that $|R(\bm{\theta}_{a},\pi(\bm{\theta}_{b}))|\leq 1$ holds and, we have equality if for all $\ell$ and $i$, $\bm{u}_{\ell,i}^{(a)}=\bm{P}_{\ell}\bm{u}_{\ell,i}^{(b)}$ and $\bm{v}_{\ell,i}^{(a)}=\bm{P}_{\ell-1}\bm{v}_{\ell,i}^{(b)}$ (proof in Section G.5). Therefore, if $R(\bm{\theta}_{a},\pi(\bm{\theta}_{b}))$ is close to one, the singular vectors of the models are well-aligned.

Figure: Figure 2: Mean and standard deviation of $R(\bm{\theta}_{a},\bm{\theta}_{b})$ from five permutation searches using WM. The red and blue bars represent the results with and without applying a permutation to $\bm{\theta}_{b}$, respectively.
Refer to caption: x2.png

Figure: (a) Evaluation results with the threshold $\gamma=0$.
Refer to caption: x3.png

Figure 2 shows the experimental results of evaluating the $R$ value between $\bm{\theta}_{a}$ and $\pi(\bm{\theta}_{b})$. A threshold $\gamma$ is introduced to examine whether the singular vectors with large singular values are preferentially aligned. For each model, we evaluate $R$ using only singular vectors whose ratio to the largest singular value is greater than $\gamma$. Thus, in the figure, $\gamma=0$ corresponds to the results when all singular vectors are used, and $\gamma=0.3$ corresponds to the results when only singular vectors with a ratio to the largest singular value exceeding 0.3 are used. When we calculated $R$, its denominator $\sum_{\ell}n_{\ell}$ was also adjusted according to the value of $\gamma$. The details of the calculation of $R$ are described in Appendix B.

The figure shows that the directions of the singular vectors are aligned with WM. Without WM, the value of $R$ is almost zero, indicating that the singular vectors are nearly orthogonal. Additionally, focusing on the difference in $\gamma$, when the singular vectors are aligned using WM, the value of $R$ is clearly larger when $\gamma$ is 0.3. This indicates that WM aligns singular vectors with larger singular values more closely. Although the value of $R$ is not necessarily very large, especially around 0.2 at most for VGG11 and ResNet20, this alignment of singular vectors still affects the merged models.

Figure 3 shows the evaluation results of $R$ between the merged model (i.e., $(\bm{\theta}_{a}+\pi(\bm{\theta}_{b}))/2$) and the pre-merged models (i.e., $\bm{\theta}_{a}$ and $\pi(\bm{\theta}_{b})$). To investigate how well the directions of singular vectors with large values are aligned between the merged and pre-merged models, we also show the results for $\gamma=0.3$ in Figure 3. The figures show that when $\gamma=0$, the value of $R$ does not change regardless of the use of WM. However, when $\gamma=0.3$, the value of $R$ changes significantly depending on whether WM is used. For example, the MLP results show that the value of $R$ exceeds 0.8 when using WM. This result indicates that the directions of singular vectors with particularly large singular values are better aligned between these models.

###### Theorem 4.1 .

### 4.4 Importance of Singular Vectors in LMC

In the previous section, we mentioned that WM aligns the directions of singular vectors with large singular values. This section clarifies why these singular vectors, rather than $L_{2}$ distances, play a crucial role in establishing LMC.

To explain this, we first focus on the difference between outputs at the $\ell$-th layers of two models, $\bm{W}_{\ell}^{(a)}$ and $\bm{W}_{\ell}^{(b)}$, given the same input $\bm{z}$ (e.g., $\bm{z}=\bm{z}_{\ell-1}^{(a)}$ or $\bm{z}=\bm{z}_{\ell-1}^{(b)}$). Suppose that the distributions of the singular values of the two weights are equal (this assumption holds for models trained with SGD, as shown in Figure 7). The difference between the outputs can be bounded from above:

$$ $\displaystyle\mathbb{E}\lVert\sigma(\bm{W}_{\ell}^{(a)}\bm{z})-\sigma(\bm{W}_{ \ell}^{(b)}\bm{z})\rVert\leq C\mathbb{E}\lVert\bm{W}_{\ell}^{(a)}\bm{z}-\bm{W} _{\ell}^{(b)}\bm{z}\rVert,$ (7) $$

where $\sigma$ is a Lipschitz continuous activation function with a constant $C>0$ (e.g., $C=1$ for the ReLU function). From Equation 7, we can see that depending on the distribution of the input $\bm{z}$, the outputs of the two layers can be close even when the distance between the two weights is not.

Let $\bm{W}_{\ell}^{(a)}=\sum_{i}\bm{u}_{\ell,i}^{(a)}s_{i}^{(a)}(\bm{v}_{\ell,i}^{
(a)})^{\top}$ and $\bm{W}_{\ell}^{(b)}=\sum_{i}\bm{u}_{\ell,i}^{(b)}s_{i}^{(b)}(\bm{v}_{\ell,i}^{
(b)})^{\top}$ be the SVDs of their weights. Here, we assume that, for some index $k$, the direction of $\bm{z}$ is always in the direction of the $k$-th right singular vector $\bm{v}_{\ell,k}^{(a)}$ with $\bm{W}^{(a)}_{\ell}$. Then, the product between $\bm{W}_{\ell}^{(a)}$ and $\bm{z}$ is given by $\bm{W}_{\ell}^{(a)}\bm{z}=\sum_{i}\bm{u}_{\ell,i}^{(a)}s_{i}^{(a)}(\bm{v}_{
\ell,i}^{(a)})^{\top}\bm{z}=\bm{u}_{\ell,k}^{(a)}s_{k}^{(a)}(\bm{v}_{\ell,k}^{
(a)})^{\top}\bm{z}$ because the singular vectors are orthogonal. Therefore, Equation 7 can be rewritten as

$$ $\displaystyle\mathbb{E}\lVert\sigma(\bm{W}_{\ell}^{(a)}\bm{z})-\sigma(\bm{W}_{ \ell}^{(b)}\bm{z})\rVert\leq C\mathbb{E}\Big{\lVert}\bm{u}_{\ell,k}^{(a)}s_{k} ^{(a)}\bm{v}_{\ell,k}^{(a)}\bm{z}-\sum_{i}\bm{u}_{\ell,i}^{(b)}s_{i}^{(b)}\bm{ v}_{\ell,i}^{(b)}\bm{z}\Big{\rVert}.$ (8) $$

Thus, as long as the directions of the $k$-th singular vectors of the two weights are aligned (i.e., $\bm{v}_{\ell,k}^{(a)}=\bm{v}_{\ell,k}^{(b)}$ and $\bm{u}_{\ell,k}^{(a)}=\bm{u}_{\ell,k}^{(b)}$ hold), the outputs of the two layers will coincide regardless of the other singular vectors. Note that the $L_{2}$ distance between the two weights is not necessarily close to zero since the directions of the other singular vectors need not be aligned. In fact, in Appendix C, we provide an example where the $L_{2}$ distance is not close to zero even though the output is zero.

More generally, the following theorem holds for the difference between the outputs of the two layers.

###### Theorem 4.2 .

For the difference, we have

$$ $\mathbb{E}\|\sigma(\bm{W}^{(a)}_{\ell}\bm{z})-\sigma(\bm{W}^{(b)}_{\ell}\bm{z} )\|\leq C\bigg{(}\sum_{i}(s_{\ell,i}^{(a)})^{2}\mathbb{E}((\bm{v}_{\ell,i}^{(a )})^{\top}\bm{z})^{2}+\sum_{i}(s_{\ell,i}^{(b)})^{2}\mathbb{E}((\bm{v}_{\ell,i }^{(b)})^{\top}\bm{z})^{2}\\ -2\sum_{i,j}s_{\ell,i}^{(a)}s_{\ell,j}^{(b)}(\bm{u}_{\ell,i}^{(a)})^{\top}\bm{ u}_{\ell,j}^{(b)}\mathbb{E}(\bm{v}_{\ell,i}^{(a)})^{\top}\bm{z}(\bm{v}_{\ell,j }^{(b)})^{\top}\bm{z}\bigg{)}^{1/2}.$ (9) $$

The proof of Theorem 4.2 is provided in Section G.6.
If the right-hand side of Equation 9 in the theorem is small, the difference is also small. Note that each sum on the right-hand side includes the inner product between the right singular vector and the input (i.e., $(\bm{v}_{\ell,i}^{(a)})^{\top}\bm{z}$ and $(\bm{v}_{\ell,i}^{(b)})^{\top}\bm{z}$). Since singular vectors are orthogonal to each other, if $\bm{z}$ is aligned with one singular vector, its inner product with the other singular vectors will be small. In other words, right singular vectors with a large inner product with the input determine the difference between the outputs of the two layers.

Figure: Figure 4: Average absolute values of the inner products of the right singular vectors and the input of the second layer. The figure shows results for ten models trained with different seeds, each represented by a different color. The test dataset is used as input for the models. In each plot, the vertical axis denotes the value of $\mathbb{E}(\bm{v}_{\ell,i}^{\top}\bm{z}_{\ell-1})^{2}$, and the horizontal axis denotes the index $i$ of the right singular vector. The left side of each plot corresponds to singular vectors with large singular values. The results for all layers are shown in Figure 8.
Refer to caption: x5.png

In the context of WM, it is desirable that the input vector has the same direction as the right singular vectors with large singular values because WM preferentially aligns these singular vectors of the two weights. To verify this, we experimentally investigate the relationship between the directions of the right singular vectors and that of the input vector. Figure 4 shows the value of $\mathbb{E}(\bm{v}_{\ell,i}^{\top}\bm{z}_{\ell-1})^{2}$ for the $i$-th right singular vector $\bm{v}_{\ell,i}$ in the second layer (i.e., $\ell=2$) and the corresponding hidden layer input $\bm{z}_{\ell-1}$ for each model. The results show that in the hidden layer, the singular vectors with large singular values have large inner products with the input vectors, which indicates that the permutations found by WM make LMC more feasible. In particular, Figure 3 shows that by aligning the directions of the singular vectors between the two models $\bm{\theta_{a}}$ and $\bm{\theta_{b}}$ using WM, the directions of the singular vectors with large singular values of the models before and after merging (e.g., $\bm{\theta}_{a}$ and $(\bm{\theta}_{a}+\bm{\theta}_{b})/2$) are well aligned. This suggests that the hidden layer outputs of the models before and after merging are closer, which contributes to the establishment of LMC.

Some studies (Ainsworth et al., ; Entezari et al., ) have empirically observed that increasing the model width makes it easier to satisfy LMC using WM. In Section H.3, we provide an analysis to explain this observation in terms of singular-vector alignment. Furthermore, Qu & Horvath () showed that strengthening weight decay and increasing the learning rate make it easier for LMC to be established through WM. Section H.4 demonstrates that increasing these values reduces the proportion of large singular values in the weights of each layer, facilitating the alignment of the corresponding singular vectors through WM, and thus making LMC easier to achieve.

###### Theorem 4.2 .

## 5 Activation Matching

Ainsworth et al. () proposed activation matching (AM) as a permutation search method different from WM. This section compares AM and WM, and explains that their results are almost similar.

Activation Matching (AM) searches for a permutation $\pi^{*}$ based on the following equation:

$$ $\displaystyle\pi^{*}=\operatorname*{arg\,min}_{\pi=(\bm{P}_{\ell})_{\ell}}\sum _{\ell\in[L]}\mathbb{E}\|\bm{z}^{(a)}_{\ell}-\bm{P}_{\ell}\bm{z}^{(b)}_{\ell} \|^{2}.$ (10) $$

Unlike WM, AM can be solved as a simple linear sum assignment problem because it can be optimized independently for each layer, allowing for the optimal solution to be obtained.

The minimization of Equation 10 is closely related to Theorem 4.2. Specifically, in a permutation search for the $\ell$-th layer, if we assume that the outputs $\bm{z}_{\ell-1}^{(a)}$ and $\bm{z}_{\ell-1}^{(b)}$ from the previous layer are sufficiently close under the permutation $\bm{P}_{\ell-1}$ (i.e., $\bm{z}_{\ell-1}^{(a)}\approx\bm{P}_{\ell-1}\bm{z}_{\ell-1}^{(b)}$), then minimizing the right-hand side of Equation 9 becomes equivalent to reducing the objective function in Equation 10. In other words, similar to WM, AM may search for permutations that align the singular vectors with large singular values between two models. To verify this, the results of model merging using AM are presented in Table 3. The experimental settings are the same as those used for WM in Section 3.2. Additionally, to evaluate how well the singular vectors align through permutation, the $R$ calculation results are shown in Figures 14 and 15. These results closely resemble those for WM in Tables 1, 2 and 3, suggesting that the reason AM achieves LMC is likely the same as for WM.

## 6 Comparison with Straight-Through Estimator

**Table 2: Results of model merging with STE.**
| Dataset | Network | Barrier ($\lambda=1/2$) | $L_{2}$ dist. w/o STE | $L_{2}$ dist. w/ STE | $R(\bm{\theta}_{a},\pi(\bm{\theta}_{b}))$ ($\gamma=0.3)$ |
| --- | --- | --- | --- | --- | --- |
| CIFAR10 | VGG11 | $0.06\pm 0.042$ | $799.503\pm 16.396$ | $799.779\pm 16.177$ | $0.036\pm 0.007$ |
| ResNet20 | $0.119\pm 0.119$ | $710.762\pm 16.261$ | $711.142\pm 16.048$ | $0.013\pm 0.005$ |  |
| FMNIST | MLP | $-0.342\pm 0.066$ | $121.853\pm 5.83$ | $118.316\pm 5.453$ | $0.081\pm 0.008$ |
| MNIST | MLP | $-0.037\pm 0.008$ | $81.231\pm 5.58$ | $73.994\pm 5.58$ | $0.211\pm 0.013$ |

This section discusses the relationship between the straight-through estimator (STE), a more direct method for finding permutations, and WM in terms of singular vectors. STE uses a dataset to find permutation matrices with a small barrier value. This section also explains that STE and WM are based on fundamentally different principles, and shows how this difference significantly impacts LMC among three or more models.

### 6.1 Straight-through Estimator (STE)

Ainsworth et al. () proposed the STE, which finds a permutation $\pi$ such that

$$ $\displaystyle\operatorname*{arg\,min}_{\pi}\mathcal{L}\bigl{(}\left(\bm{\theta }_{a}+\pi(\bm{\theta}_{b})\right)/2\bigr{)}.$ (11) $$

Since Equation 11 is difficult to solve directly, Ainsworth et al. () proposed a method to approximate the solution. Later, Peña et al. () proposed a method to solve Equation 11 directly using Sinkhorn’s algorithm. We adopt the latter method based on Sinkhorn’s algorithm, which we refer to as STE in this paper.

### 6.2 Experimental Results of Model Merging by STE

Figure: Figure 5: Evaluation results of $R$ between each pair of the models with $\gamma=0.3$.
Refer to caption: x6.png

Figure: (a) Results of WM.
Refer to caption: x7.png

The experimental results of model merging using STE are shown in Table 2. This table also shows the $L_{2}$ distance and the $R$ value between the two models before and after the permutation. Despite the relatively small barrier value, Table 2 shows that the $L_{2}$ distance between the two models before and after permutation hardly changes compared to the results with WM shown in Table 1. Since $R(\bm{\theta}_{a},\pi(\bm{\theta}_{b}))$ is nearly zero, the singular vectors between the two models are likely not aligned at all. Therefore, the reason for satisfying LMC by STE is completely different from that of WM.

### 6.3 LMC among Three Models

The previous subsection shows that the permutation matrices found by STE do not align the directions of the singular vectors of the models. This suggests that STE finds a permutation that reduces the loss of the merged model based on the loss landscape rather than the linear algebraic properties of the weight matrices of each layer. The difference between the principles of STE and WM could result in a qualitative difference in LMC among three or more models.

Suppose we have three SGD solutions: $\bm{\theta}_{a}$, $\bm{\theta}_{b}$, and $\bm{\theta}_{c}$. Let $\pi_{b}$ and $\pi_{c}$ be permutations that satisfy LMC between $\bm{\theta}_{a}$ and $\pi_{b}(\bm{\theta}_{b})$, and $\bm{\theta}_{a}$ and $\pi_{c}(\bm{\theta}_{c})$, respectively. If permutations found by STE depend on the locality of the loss landscape rather than the linear algebraic properties of the model weights, there is no guarantee that $\pi_{b}(\bm{\theta}_{b})$ and $\pi_{c}(\bm{\theta}_{c})$ are linearly mode-connected. In contrast, permutations found by WM align the directions of the singular vectors of the two models. This means that the singular vectors of $\pi_{b}(\bm{\theta}_{b})$ and $\pi_{c}(\bm{\theta}_{c})$ are also expected to be aligned. Thus, the LMC between $\pi_{b}(\bm{\theta}_{b})$ and $\pi_{c}(\bm{\theta}_{c})$ may not be satisfied with STE, while it is likely to be satisfied with WM.

We performed model merging experiments among three models to confirm the validity of the above discussion. First, Figure 5 presents the results of examining how well the singular vectors are aligned in each model pair by WM. Since the models $\bm{\theta}_{b}$ and $\bm{\theta}_{c}$ are matched to $\bm{\theta}_{a}$ through WM, it is expected that $R(\bm{\theta}_{a},\pi_{b}(\bm{\theta}_{b}))$ and $R(\bm{\theta}_{a},\pi_{c}(\bm{\theta}_{c}))$ would be large. On the other hand, although $\bm{\theta}_{b}$ and $\bm{\theta}_{c}$ were not explicitly aligned, $R(\pi_{b}(\bm{\theta}_{b}),\pi_{c}(\bm{\theta}_{c}))$ is clearly greater than zero, indicating that the directions of these two singular vectors are indirectly aligned by WM. From this result, the barrier between the models $\pi_{b}(\bm{\theta}_{b})$ and $\pi_{c}(\bm{\theta}_{c})$ is expected to be small. To confirm this, Figure 6 shows the test accuracy landscape around $\bm{\theta}_{a}$, $\pi_{b}(\bm{\theta}_{b})$, and $\pi_{c}(\bm{\theta}_{c})$. The results of model merging at $\lambda=1/2$ for each pair of models are also shown in Table 4. Table 4 shows the mean and standard deviation of model merging three times. As can be seen from Figure 6, the barrier between $\pi_{b}(\bm{\theta}_{b})$ and $\pi_{c}(\bm{\theta}_{c})$ is smaller with WM than with STE. This means that there is a significant difference between the principles of permutations obtained by WM and STE. Figure 6 also shows that the landscape of test accuracy is flatter around the three models with WM than with STE. Therefore, WM is likely to be more advantageous, especially for merging three or more models.

## 7 Conclusion

This paper analyzed why linear mode connectivity (LMC) is satisfied through permutation search with weight matching (WM). First, we demonstrated that WM does not reduce the distance between the weights of two models as significantly as previously thought. We then analyzed WM using singular value decomposition (SVD) and found that WM aligns the directions of singular vectors with large singular values, which plays a crucial role in achieving LMC. Additionally, we showed that the reason LMC is established in AM is the same as for WM. Finally, we discussed the difference between STE and WM from the perspective of singular vectors.

## Appendix

## Appendix A Extended Related Work

#### (Linear) mode connectivity.

Several studies (Garipov et al., ; Draxler et al., ; Freeman & Bruna, ) have found that different neural network solutions can be connected by nonlinear paths with almost no increase in loss. Nagarajan & Kolter () first discovered that solutions can be connected by linear paths with an almost constant loss value when training models on MNIST with the same random initial values. Later, Frankle et al. () demonstrated experimentally that LMC is not always satisfied between two SGD solutions, even with the same initial parameters, depending on the datasets and model architectures. However, they also showed that if a single model is trained for a certain period and then two models are trained independently from this pre-trained model as a starting point, they are linearly mode-connected. Furthermore, Frankle et al. () explored the relationship between LMC and the lottery-ticket hypothesis (Frankle & Carbin, ). Entezari et al. () conjectured that LMC is satisfied with a high probability between two SGD solutions by accounting for permutation symmetries in the hidden layers. Subsequently, Ainsworth et al. () proposed a WM method by formulating neuron alignment as a bipartite graph matching problem and solving it approximately. Later, Peña et al. () suggested using Sinkhorn’s algorithm to solve the WM directly. Some previous studies (Ainsworth et al., ; Crisostomi et al., ) have also proposed permutation search methods for achieving LMC between multiple models. However, all of these methods use WM, and no methods have been proposed that use information from the loss function, such as STE. Therefore, in this paper, we created a pair of models and performed a permutation search to clarify the differences between WM and STE in a fair manner. The investigation of permutation search methods for multiple models is a future work.

While several papers (Venturi et al., ; Nguyen et al., ; Nguyen, ; Kuditipudi et al., ) have discussed nonlinear mode connectivity, there is little theoretical analysis on LMC. Ferbach et al. () provided an upper bound on the minimal width of the hidden layer to satisfy LMC. However, to prove this, they assumed the independence of all neuron’s weight vectors inside a given layer. It is unlikely that this assumption holds for models after training. Zhou et al. () introduced the concept of layerwise linear feature connectivity (LLFC) and showed that LLFC implies LMC. Additionally, Zhou et al. () demonstrated that if weak additivity for ReLU activation and the commutativity property are satisfied, then both LLFC and LMC are satisfied. However, we show that the $L_{2}$ distance between the models after permutation is not close enough to satisfy the commutativity property. This motivated us to investigate the relationship between LMC and WM.

#### Model merging.

Relevant topics of LMC include model merging and federated learning. McMahan et al. () and Konečný et al. () introduced the concept of federated learning, where a model is trained on divided datasets. Wang et al. () proposed a federated learning method by permuting each component unit and then averaging the weights of the models. Singh & Jaggi () proposed a method for merging models by performing alignments of model weights using optimal transport, which is similar to the method proposed by Ainsworth et al. (). Although their method is designed for model fusion and its performance is inferior to that of Ainsworth et al.’s method, it can be considered an LMC-based method because it uses hard alignments for the same architecture. Wortsman et al. () proposed a method to improve test accuracy without increasing inference cost, unlike ensemble methods, by averaging the weights of models fine-tuned with different hyperparameters.

#### Low-rank bias.

Empirically, some previous studies (Tukan et al., ; Arora et al., ; Alvarez & Salzmann, ; Yu et al., ; Denton et al., ) on the compression of trained DNN models have pointed out that even when the weights are replaced with low-rank matrices, the accuracy does not decrease significantly. This suggests that SGD has an implicit bias toward reducing the model weights to low-rank. Galanti et al. () and Timor et al. () further discuss the low-rank effect on the weights of trained models introduced by weight decay and a small initialization scale. In particular, Galanti et al. () state that the weights become more low-rank by strengthening the weight decay and increasing the learning rate, and this is expected to help establish LMC via WM (which we experimentally confirm in Section H.4).

## Appendix B Calculation of R 𝑅 R italic_R with Threshold γ 𝛾 \gamma italic_γ

This section describes how to calculate the $R$ value with a threshold $\gamma>0$. Given two models, $\bm{\theta}_{a}$ and $\bm{\theta}_{b}$, let $\bm{W}_{\ell}^{(a)}$ and $\bm{W}_{\ell}^{(b)}$ be the weights of the $\ell$-th layers of the models $\bm{\theta}_{a}$ and $\bm{\theta}_{b}$, respectively. Let $\sum_{i}\bm{u}_{\ell,i}^{(a)}s_{\ell,i}^{(a)}(\bm{v}_{\ell,i}^{(a)})^{\top}$ and $\sum_{i}\bm{u}_{\ell,i}^{(b)}s_{\ell,i}^{(b)}(\bm{v}_{\ell,i}^{(b)})^{\top}$ be the SVDs of these weights. Also, let $s^{(a)}$ and $s^{(b)}$ be the maximum singular values in all the layers of the models $\bm{\theta}_{a}$ and $\bm{\theta}_{b}$, respectively. The $R$ value with the threshold $\gamma$ is calculated as follows:

$$ $\displaystyle R_{\gamma}(\bm{\theta}_{a},\bm{\theta}_{b})=\frac{\sum_{\ell,i,j }I[(s_{\ell,i}^{(a)}\geq\gamma s^{(a)})\land(s_{\ell,i}^{(b)}\geq\gamma s^{(b) })](\bm{u}^{(a)}_{\ell,i})^{\top}(\bm{u}^{(b)}_{\ell,j})(\bm{v}^{(a)}_{\ell,i} )^{\top}(\bm{v}^{(b)}_{\ell,j})}{\sum_{\ell}\min\{n_{\ell}^{(a)},n_{\ell}^{(b) }\}},$ (12) $$

where $n_{\ell}^{(a)}$ and $n_{\ell}^{(b)}$ are the numbers of singular values greater than $\gamma s^{(a)}$ and $\gamma s^{(b)}$, respectively, and $I$ is an indicator function that returns one if the given logical expression is true and zero if it is false.

Finally, we will briefly explain that $|R_{\gamma}(\bm{\theta}_{a},\bm{\theta}_{b})|\leq 1$ holds. We first define the new weight matrices by $\bm{W}^{\prime{(a)}}_{\ell}=\sum_{i}\bm{u}_{\ell,i}^{(a)}I[(s_{\ell,i}^{(a)}
\geq\gamma s^{(a)})](\bm{v}_{\ell,i}^{(a)})^{\top}$ and $\bm{W}^{\prime{(b)}}_{\ell}=\sum_{i}\bm{u}_{\ell,i}^{(b)}I[(s_{\ell,j}^{(b)}
\geq\gamma s^{(b)})](\bm{v}_{\ell,i}^{(b)})^{\top}$. From the definition, we have:

$$ $\displaystyle\operatorname{tr}\left((\bm{W}^{\prime{(a)}}_{\ell})^{\top}\bm{W} ^{\prime{(b)}}_{\ell}\right)=\sum_{i,j}I[(s_{\ell,i}^{(a)}\geq\gamma s^{(a)}) \land(s_{\ell,j}^{(b)}\geq\gamma s^{(b)})](\bm{u}^{(a)}_{\ell,i})^{\top}(\bm{u }^{(b)}_{\ell,j})(\bm{v}^{(a)}_{\ell,i})^{\top}(\bm{v}^{(b)}_{\ell,j}).$ (13) $$

Note that the $i$-th singular values of these weights $\bm{W}_{\ell}^{\prime{(a)}}$ and $\bm{W}_{\ell}^{\prime{(b)}}$ can be regarded as $I[s^{(a)}_{\ell,i}\geq\gamma s^{(a)}]$ and $I[s^{(b)}_{\ell,i}\geq\gamma s^{(b)}]$, respectively. Therefore, von Neumann’s trace inequality (von Neumann, ) yields that:

$$ $\displaystyle\operatorname{tr}\left((\bm{W}^{\prime{(a)}}_{\ell})^{\top}\bm{W} ^{\prime{(b)}}_{\ell}\right)\leq\sum_{i}I[s_{\ell,i}^{(a)}\geq\gamma s^{(a)}]I [s_{\ell,i}^{(b)}\geq\gamma s^{(b)}]=\min\{n_{\ell}^{(a)},n_{\ell}^{(b)}\}.$ (14) $$

From Equations 13 and 14, we have:

$$ $\displaystyle\sum_{i,j}I[(s_{\ell,i}^{(a)}\geq\gamma s^{(a)})\land(s_{\ell,j}^ {(b)}\geq\gamma s^{(b)})](\bm{u}^{(a)}_{\ell,i})^{\top}(\bm{u}^{(b)}_{\ell,j}) (\bm{v}^{(a)}_{\ell,i})^{\top}(\bm{v}^{(b)}_{\ell,j})\leq\min\{n_{\ell}^{(a)}, n_{\ell}^{(b)}\}.$ (15) $$

By summing both sides for $\ell$, we get $|R_{\gamma}(\bm{\theta}_{a},\bm{\theta}_{b})|\leq 1$.

## Appendix C Simple Example of Theorem 4.2

In Section 4.4, we explained that even when the $L_{2}$ distance between the weights of two models is large, their outputs can be close depending on the input distribution. Here, we use a simple example for a more detailed analysis.

Consider the weights of two models, $\bm{W}^{(a)}$ and $\bm{W}^{(b)}$, given by:

$$ $\displaystyle\bm{W}^{(a)}=\begin{pmatrix}-0.398&-0.003&0.210\\ 1.059&0.303&0.521\\ 0.609&-0.785&-0.235\end{pmatrix},\bm{W}^{(b)}=\begin{pmatrix}-0.255&-0.319&-0. 559\\ 1.031&-0.155&0.484\\ 0.742&-0.114&-0.604\end{pmatrix}.$ (16) $$

The SVDs of these matrices are represented by:

$$ $\displaystyle\bm{W}^{(a)}$ $\displaystyle=\bm{U}^{(a)}\bm{S}^{(a)}(\bm{V}^{(a)})^{\top}$ (17) $\displaystyle\approx\begin{pmatrix}-0.260&-0.127&0.957\\ 0.850&-0.501&0.165\\ 0.458&0.856&0.238\end{pmatrix}\begin{pmatrix}1.317&0&0\\ 0&0.959&0\\ 0&0&0.277\end{pmatrix}\begin{pmatrix}0.974&-0.077&0.213\\ 0.043&-0.859&-0.510\\ -0.222&-0.506&0.833\end{pmatrix}^{\top},$ (18) $$

and

$$ $\displaystyle\bm{W}^{(b)}$ $\displaystyle=\bm{U}^{(b)}\bm{S}^{(b)}(\bm{V}^{(b)})^{\top}$ (19) $\displaystyle\approx\begin{pmatrix}-0.261&0.587&0.767\\ 0.850&-0.237&0.470\\ 0.458&0.774&-0.437\end{pmatrix}\begin{pmatrix}1.317&0&0\\ 0&0.958&0\\ 0&0&0.277\end{pmatrix}\begin{pmatrix}0.974&-0.077&0.213\\ 0.188&-0.249&-0.950\\ -0.126&-0.965&0.228\end{pmatrix}^{\top},$ (20) $$

respectively.

In this case, the distance between these weights is $\|\bm{W}^{(a)}-\bm{W}^{(b)}\|\approx 1.236$. On the other hand, if the input vector $\bm{z}$ is given by $\bm{z}=k\begin{pmatrix}0.974&-0.077&0.213\end{pmatrix}$, where $k$ is an arbitrary (but not too large) real scalar value, then $\|\sigma(\bm{W}^{(a)}\bm{z})-\sigma(\bm{W}^{(b)}\bm{z})\|\approx 0$ holds.

## Appendix D Experimental Setup

This section describes the experimental setup for training neural networks to obtain SGD solutions. We apply Sinkhorn’s algorithm for permutation based on WM and STE. Thus, we also provide detailed information on the experimental setup for Sinkhorn’s algorithm. Four datasets were used in this study: MNIST (Lecun et al., ), Fashion-MNIST (FMNIST) (Xiao et al., ), CIFAR10 (Krizhevsky et al., ), and ImageNet (Deng et al., ).

All experiments were conducted on a Linux workstation with two AMD EPYC 7543 32-Core processors, eight NVIDIA A30 GPUs, and 512 GB of memory. The PyTorch 2.1.0(^2^22[https://pytorch.org/](https://pytorch.org/)), PyTorch Lightning 2.1.0(^3^33[https://lightning.ai/docs/pytorch/stable/](https://lightning.ai/docs/pytorch/stable/)), and torchvision 0.16.0(^4^44[https://pytorch.org/vision/stable/index.html](https://pytorch.org/vision/stable/index.html)) libraries were used for model training and evaluation.

### D.1 Model Training

#### MLP on MNIST and FMNIST.

Following the settings in (Ainsworth et al., ), we trained a Multi-Layer Perceptron (MLP) with three hidden layers, each comprising 512 units. The hidden layers use the ReLU function as their activation function. For the MNIST and FMNIST datasets, we optimized using the Adam algorithm with a learning rate of $1\times 10^{-3}$. The batch size and maximum number of epochs were set to 512 and 100, respectively.

#### VGG11 and ResNet20 on CIFAR10.

We utilized the VGG16 and ResNet20 architectures of (Ainsworth et al., ). To accomplish Layer-wise Model Compression (LMC), we increased the widths of VGG11 and ResNet20 by factors of 4 and 16, respectively. As described in (Jordan et al., ), we used the training dataset to repair the BatchNorm layers in these models during model merging. Optimization was conducted using Adam with a learning rate of $1\times 10^{-3}$. The batch size and maximum number of epochs were set to 512 and 100, respectively. The following data augmentations were performed during training: random $32\times 32$ pixel crops, and random horizontal flips.

#### ResNet50 on ImageNet.

ResNet50 models were trained using a training script published on GitHub(^5^55[https://github.com/libffcv/ffcv-imagenet/tree/main](https://github.com/libffcv/ffcv-imagenet/tree/main)) by the FFCV library (Leclerc et al., ). The ”rn50_40_epochs.yaml” file in the repository was used for the training setup. In the file, we changed use_blurpool to “0”. As described in (Jordan et al., ), we repaired the BatchNorm layers in these models during model merging by using the training dataset. Since ImageNet is a large dataset, we used 50,000 randomly selected images from the training set to repair the batch normalization layers.

### D.2 Permutation Search

For permutation search in WM and STE, we employed the method based on Sinkhorn’s algorithm as proposed by Peña et al. (). We utilized the implementation provided by the authors in their GitHub repository(^6^66[https://github.com/fagp/sinkhorn-rebasin](https://github.com/fagp/sinkhorn-rebasin)). DistL2Loss and MidLoss were used as loss functions for permutation searches corresponding to WM and STE, respectively. Optimization was performed using Adam with a learning rate of 1 for MLP, VGG11, and ResNet20 and 10 for ResNet50, setting the maximum number of epochs to 10 for DistL2Loss and five for MidLoss. For MidLoss, the batch size was set to 512; for DistL2Loss, there was no batch size because the dataset was not used. 100 iterations of parameter updates were performed per epoch for DistL2Loss.

For activation matching (AM), the permutation search is divided for each layer, so the optimal solution can be obtained efficiently. We implemented AM-based permutation search following the GitHub repository released by Ainsworth et al. ()(^7^77[https://github.com/samuela/git-re-basin](https://github.com/samuela/git-re-basin)). In this paper, the linear_sum_assignment function of Scipy (Virtanen et al., ) was used for the permutation search in AM.

## Appendix E Discussion on Commutativity Property

Zhou et al. () show that LMC is satisfied if weak additivity for ReLU activations and commutativity hold. Given two models, $\bm{\theta}_{a}$ and $\bm{\theta}_{b}$, commutativity between them is satisfied if for all layers $\ell\in[L]$, $\bm{W}_{\ell}^{(a)}\bm{z}_{\ell-1}^{(a)}+\bm{W}_{\ell}^{(b)}\bm{z}_{\ell-1}^{(
b)}=\bm{W}_{\ell}^{(a)}\bm{z}_{\ell-1}^{(b)}+\bm{W}_{\ell}^{(b)}\bm{z}_{\ell-1
}^{(a)}$ holds, where $L$ is the number of layers of the models(^8^88Strictly speaking, given a data distribution $\mathcal{D}$, the commutativity property is satisfied if the equation almost surely holds for $\mathcal{D}$. This definition is equivalent to Zhou et al. (), although it differs slightly.). The commutativity property can be rewritten as $\forall\ell\in[L];(\bm{W}_{\ell}^{(a)}-\bm{W}_{\ell}^{(b)})(\bm{z}_{\ell-1}^{(
a)}-\bm{z}_{\ell-1}^{(b)})=\bm{0}$. Therefore, Zhou et al. () in Section 5.2 justify the WM-based permutation search method because WM aims to minimize Equation 2, which corresponds to the first factor in the equation. However, as shown in our paper, WM only slightly reduces the distance between the two models, contradicting their claim.

Appendix B.5 of (Zhou et al., ) explains in a different way why the commutativity property is satisfied in WM. Specifically, they consider a stronger form of the commutativity property:

$$ $\displaystyle\forall\ell\in[L];\bm{W}_{\ell}^{(a)}\bm{z}_{\ell-1}^{(a)}=\bm{W} _{\ell}^{(b)}\bm{z}_{\ell-1}^{(a)}\land\bm{W}_{\ell}^{(b)}\bm{z}_{\ell-1}^{(b) }=\bm{W}_{\ell}^{(a)}\bm{z}_{\ell-1}^{(b)}.$ (21) $$

To ensure this stronger form, we need to find $\bm{P}_{\ell}$ and $\bm{P}_{\ell-1}$ such that:

$$ $\displaystyle(\bm{W}_{\ell}^{(a)}-\bm{P}_{\ell}\bm{W}_{\ell}^{(b)}\bm{P}^{\top }_{\ell-1})\bm{z}_{\ell-1}^{(a)}=\bm{0}\land(\bm{P}_{\ell}\bm{W}_{\ell}^{(b)} \bm{P}^{\top}_{\ell-1}-\bm{W}_{\ell}^{(a)})\bm{P}_{\ell-1}\bm{z}_{\ell-1}^{(b) }=\bm{0}.$ (22) $$

They argue that the commutativity property easily holds because it is easy to find permutation matrices $\bm{P}_{\ell}$ and $\bm{P}_{\ell-1}$ that satisfy Equation 22 due to the small actual dimension of the hidden layer inputs $\bm{z}_{\ell-1}^{(a)}$ and $\bm{z}_{\ell-1}^{(b)}$ (i.e., these vectors are biased in a particular direction).

However, several points need to be addressed regarding this explanation. First, if Equation 21 holds, then LMC is satisfied without any assumptions, such as the commutativity property or weak additivity of ReLU activations. Since this equation must hold for all layers, it must also hold for the input layer where $\bm{z}_{\ell-1}^{(a)}=\bm{z}_{\ell-1}^{(b)}=\bm{x}$. In that case, the outputs of the input layers are equivalent between the two models. The same holds for subsequent layers, so the outputs of the two models are the same in all hidden layers. Therefore, the outputs of the hidden layers of the merged model must also be identical to those of the pre-merged models in all layers. Thus, LMC obviously holds. In other words, if the reason of the establishment of LMC is that Equation 21 holds, then the essential reason for the establishment of LMC is that WM makes the outputs of the hidden layers of the two models close, suggesting that our argument in this paper is more fundamental in establishing LMC.

Second, the small actual dimension of the hidden layer inputs $\bm{z}_{\ell-1}^{(a)}$ and $\bm{z}_{\ell-1}^{(b)}$ does not necessarily mean that Equation 22 is easier to satisfy by finding permutations that minimize Equation 2. As shown in Section 4.3, WM preferentially aligns the directions of singular vectors with large singular values, while other singular vectors are difficult to align. If the hidden layer inputs $\bm{z}_{\ell-1}^{(a)}$ and $\bm{z}_{\ell-1}^{(b)}$ are not oriented in the same directions as the right singular vectors with large singular values, then WM will not help satisfy Equation 22. Zhou et al. () did not mention this second point. On the other hand, we analyzed this point in Section 4.4.

## Appendix F Convolutional Layers

This section discusses a theorem similar to Theorem 4.1 for convolutional neural networks (CNNs).

### F.1 Notation

We introduce the notation used in the following sections. Each element of a tensor is specified by a simple italic variable with subscripts. For example, for a third-order tensor $\bm{X}$, its $i,j,k$-th component is denoted by $X_{i,j,k}$. We also use Python-like slice notation. For example, $\bm{X}_{1,:}$ denotes the first row of the matrix $\bm{X}$. For a complex matrix $\bm{X}$, let $\bm{X}^{*}=\overline{\bm{X}}^{\top}$ be its unitary transpose, where $\overline{\bm{X}}$ denotes the complex conjugate of $\bm{X}$.

### F.2 Matrix Representation of Convolutional Layer

This subsection introduces the matrix representation of a convolutional layer. Let $\bm{X}\in\mathbb{R}^{m\times n\times n}$ and $\bm{Y}\in\mathbb{R}^{m\times n\times n}$ be the input and the output of the $\ell$-th convolutional layer, respectively. Here, $m$ denotes the number of input and output channels and $n$ denotes the size of the height and width of the input. For simplicity, we assume that the numbers of output channels and input channels are identical, as well as the sizes of the height and width of the input, although our analysis is applicable even when they are not. Let $\bm{K}\in\mathbb{R}^{n\times n\times m\times m}$ be the kernel of the $\ell$-th layer. Then, for the $c,r,i$-th element of the output $\bm{Y}$ is given by

$$ $\displaystyle Y_{c,r,i}=\sum_{d\in[m],p\in[n],q\in[n]}X_{d,r+p,i+q}K_{p,q,c,d}.$ (23) $$

There exists a matrix $\bm{M}$ such that $\mathrm{vec}(\bm{Y})=\bm{M}\mathrm{vec}(\bm{X})$ holds (Sedghi et al., ; Jain, ; Goodfellow et al., ), where

$$ $\displaystyle\bm{M}=\begin{pmatrix}\bm{B}_{1,1}&\bm{B}_{1,2}&\dots&\bm{B}_{1,m }\\ \bm{B}_{2,1}&\bm{B}_{2,2}&\dots&\bm{B}_{2,m}\\ \vdots&\vdots&\ddots&\vdots\\ \bm{B}_{m,1}&\bm{B}_{m,2}&\dots&\bm{B}_{m,m}\end{pmatrix}.$ (24) $$

Here, for all $c,d\in[m]$, $\bm{B}_{c,d}$ is a doubly circulant matrix defined by

$$ $\displaystyle\bm{B}_{c,d}=\begin{pmatrix}\mathrm{circ}(K_{1,:,c,d})&\mathrm{ circ}(K_{2,:,c,d})&\dots&\mathrm{circ}(K_{n,:,c,d})\\ \mathrm{circ}(K_{n,:,c,d})&\mathrm{circ}(K_{1,:,c,d})&\dots&\mathrm{circ}(K_{n -1,:,c,d})\\ \vdots&\vdots&\ddots&\vdots\\ \mathrm{circ}(K_{2,:,c,d})&\mathrm{circ}(K_{3,:,c,d})&\dots&\mathrm{circ}(K_{1 ,:,c,d})\end{pmatrix},$ (25) $$

where $\mathrm{circ}$ is a function to generate a circulant matrix from a given vector. For example, given a vector $\bm{a}=(a_{1},a_{2},\dots,a_{3})$, the circulant matrix generated by $\bm{a}$ is given by $\mathrm{circ}(\bm{a})=\begin{pmatrix}a_{1}&a_{2}&a_{3}\\
a_{3}&a_{1}&a_{2}\\
a_{2}&a_{3}&a_{1}\end{pmatrix}$.

### F.3 Singular Value Decomposition and Weight Matching of Convolutional Layers

Since Equation 24 represents the matrix form of the convolutional layer, we can reach a conclusion similar to Theorem 4.1 by performing a singular value decomposition (SVD) on it. However, this matrix is very large, with a size of $mn^{2}\times mn^{2}$, making direct SVD impractical. Therefore, we decompose it into a more SVD-friendly form using a Fourier transform. Using the imaginary unit as $\eta=\sqrt{-1}$, and setting $\omega=e^{-2\pi\eta/n}$, a one-dimensional Fourier transform matrix $\bm{F}$ is defined by $F_{i,j}=(\omega^{(i-1)(j-1)})_{i,j}$.(^9^99Usually, the alphabet letters $i$ or $j$ are used for the imaginary unit, but since they are used as indices here, we use $\eta$.) A matrix for the two-dimensional Fourier transform can be defined as $\bm{Q}=(\bm{F}\otimes\bm{F})/n$. Here, $\otimes$ denotes the Kronecker product. By using this two-dimensional Fourier transform matrix $\bm{Q}$, the matrix $\bm{M}$ can be decomposed as follows:

$$ $\displaystyle\bm{M}=(\bm{I}_{m}\otimes\bm{Q})^{*}\bm{L}(\bm{I}_{m}\otimes\bm{Q }),$ (26) $$

where $\bm{I}_{m}$ denotes the identity matrix of size $m\times m$. We then have

$$ $\displaystyle\bm{L}=\begin{pmatrix}\bm{D}_{1,1}&\bm{D}_{1,2}&\dots&\bm{D}_{1,m }\\ \bm{D}_{2,1}&\bm{D}_{2,2}&\dots&\bm{D}_{2,m}\\ \vdots&\vdots&\ddots&\vdots\\ \bm{D}_{m,1}&\bm{D}_{m,2}&\dots&\bm{D}_{m,m}\end{pmatrix}.$ (27) $$

Here, for all $c,d\in[m]$, $\bm{D}_{c,d}=\bm{Q}\bm{B}_{c,d}\bm{Q}^{*}$ is a complex diagonal matrix (Sedghi et al., ). Let $\bm{G}_{:,:,w}$ be a matrix formed by extracting the $w$-th diagonal element of each diagonal matrix $\bm{D}_{c,d}$ and arranging them (i.e., $G_{c,d,w}=(\bm{D}_{c,d})_{w,w}$). Then, the following theorem holds:

###### Theorem F.1 (SVD of convolutional layer) .

Let $s_{w,i}$, $\bm{u}_{w,i}$, and $\bm{v}_{w,i}$ be the $i$-th singular value, left singular vector, and right singular vector of $\bm{G}_{:,:,w}$, respectively. Then, the matrix $\bm{M}$ representing the convolutional layer can be decomposed as follows:

$$ $\displaystyle\bm{M}=\sum_{w,i}(\bm{u}_{w,i}\otimes\bm{Q}^{*}\bm{e}_{w})s_{w,i} (\bm{v}_{w,i}\otimes\bm{Q}^{*}\bm{e}_{w})^{*},$ (28) $$

where $\bm{e}_{w}$ represents the orthonormal basis in Euclidean space $\mathbb{R}^{n^{2}}$, and $s_{w,i}$, $\bm{u}_{w,i}\otimes\bm{Q}^{*}\bm{e}_{w}$, and $\bm{v}_{w,i}\otimes\bm{Q}^{*}\bm{e}_{w}$ are the singular value, left singular vector, and right singular vector of $\bm{M}$, respectively.

The proof is shown in Section G.3. From this theorem, the following theorem can be proved:

###### Theorem F.2 .

Let $\bm{M}^{(a)}$ and $\bm{M}^{(b)}$ be the matrix representations of convolutional layers of two CNNs. From Theorem F.1, their SVDs are given by $\bm{M}^{(a)}=\sum_{w,i}(\bm{u}_{w,i}^{(a)}\otimes\bm{Q}^{*}\bm{e}_{w})s_{w,i}^
{(a)}(\bm{v}_{w,i}^{(a)}\otimes\bm{Q}^{*}\bm{e}_{w})^{*}$ and $\bm{M}^{(b)}=\sum_{w,i}(\bm{u}_{w,i}^{(b)}\otimes\bm{Q}^{*}\bm{e}_{w})s_{w,i}^
{(b)}(\bm{v}_{w,i}^{(b)}\otimes\bm{Q}^{*}\bm{e}_{w})^{*}$, respectively. Then, the WM between $\bm{M}^{(a)}$ and $\bm{M}^{(b)}$ is equivalent to finding permutation matrices $\bm{P}_{\ell}$ and $\bm{P}_{\ell-1}$ such that

$$ $\displaystyle\operatorname*{arg\,max}_{\bm{P}_{\ell},\bm{P}_{\ell-1}}\Re\sum_{ w,i,j}s_{w,i}^{(a)}s_{w,j}^{(b)}\overline{(\bm{u}_{w,i}^{(a)})^{*}(\bm{P}_{ \ell}\bm{u}_{w,j}^{(b)})}(\bm{v}_{w,i}^{(a)})^{*}(\bm{P}_{\ell-1}\bm{v}_{w,j}^ {(b)}),$ (29) $$

where $\Re z$ is the real part of $z$ for a complex number $z$.

The proof is shown in Section G.4. Similar to the case of MLP (Theorem 4.1), Theorem F.2 indicates that WM has the effect of aligning the directions of the corresponding singular vectors in convolutional layers.

###### Theorem F.1 (SVD of convolutional layer) .

###### Theorem F.2 .

## Appendix G Proofs

### G.1 Proof of Theorem 3.1

###### Proof.

From the assumption and Taylor theorem centered at $\bm{\theta}_{a}$, we have

$$ $\displaystyle\mathcal{L}(\bm{\theta}_{c})$ $\displaystyle=\mathcal{L}(\bm{\theta}_{a})+(\bm{\theta}_{c}-\bm{\theta}_{a}) \nabla\mathcal{L}(\bm{\theta}_{a})+\frac{1}{2}(\bm{\theta}_{c}-\bm{\theta}_{a} )^{\top}\bm{H}_{a}(\bm{\theta}_{c}-\bm{\theta}_{a})+O(\|\bm{\theta}_{c}-\bm{ \theta}_{a}\|^{3})$ (30) $\displaystyle=\mathcal{L}(\bm{\theta}_{a})+(1-\lambda)\beta\bm{\mu}\nabla \mathcal{L}(\bm{\theta}_{a})+\frac{1}{2}(1-\lambda)^{2}\beta^{2}\bm{\mu}^{\top }\bm{H}_{a}\bm{\mu}+O(\beta^{3}).$ (31) $$

Similarly, using Taylor theorem centered at $\pi(\bm{\theta}_{b})$, we get

$$ $\displaystyle\mathcal{L}(\bm{\theta}_{c})$ $\displaystyle=\mathcal{L}(\pi(\bm{\theta}_{b}))+(\bm{\theta}_{c}-\pi(\bm{ \theta}_{b}))\nabla\mathcal{L}(\pi(\bm{\theta}_{b}))+\frac{1}{2}(\bm{\theta}_{ c}-\pi(\bm{\theta}_{b}))^{\top}\bm{H}_{a}(\bm{\theta}_{c}-\pi(\bm{\theta}_{b}) )+O(\|\bm{\theta}_{c}-\pi(\bm{\theta}_{b})\|^{3})$ (32) $\displaystyle=\mathcal{L}(\pi(\bm{\theta}_{b}))-\lambda\beta\bm{\mu}\nabla \mathcal{L}(\pi(\bm{\theta}_{b}))+\frac{1}{2}\lambda^{2}\beta^{2}\bm{\mu}^{ \top}\bm{H}_{a}\bm{\mu}+O(\beta^{3}).$ (33) $$

Combining these equations, the barrier can be obtained as

$$ $\displaystyle B(\bm{\theta}_{a},\pi(\bm{\theta}_{b}))$ $\displaystyle=\max_{\lambda}\left(\mathcal{L}(\bm{\theta}_{c})-\lambda\mathcal {L}(\bm{\theta}_{a})-(1-\lambda)\mathcal{L}(\pi(\bm{\theta}_{b}))\right)$ (34) $\displaystyle=\max_{\lambda}\left(\lambda(\mathcal{L}(\bm{\theta}_{c})- \mathcal{L}(\bm{\theta}_{a}))+(1-\lambda)(\mathcal{L}(\bm{\theta}_{c})- \mathcal{L}(\pi(\bm{\theta}_{b})))\right)$ (35) $\displaystyle=\max_{\lambda}\Big{(}\beta\lambda(1-\lambda)\bm{\mu}^{\top}( \nabla\mathcal{L}(\bm{\theta}_{a})-\nabla\mathcal{L}(\pi(\bm{\theta}_{b})))$ (36) $\displaystyle\hskip 50.00008pt+\frac{1}{2}\beta^{2}\lambda(1-\lambda)\bm{\mu}^ {\top}\left((1-\lambda)\bm{H}_{a}+\lambda\bm{H}_{b}\right)\bm{\mu}\Big{)}+O( \beta^{3}).$ (37) $$

∎

###### Proof.

### G.2 Proof of Theorem 4.1

###### Proof.

Consider the $L_{2}$ norm of the $\ell$-th layer $\|\bm{W}_{\ell}^{(a)}-\bm{P}_{\ell}\bm{W}_{\ell}^{(b)}\bm{P}_{\ell-1}^{\top}\|
^{2}$. Using the fact that the $L_{2}$ norm can be rewritten using trace, we have

$$ $\displaystyle\|\bm{W}_{\ell}^{(a)}-\bm{P}_{\ell}\bm{W}_{\ell}^{(b)}\bm{P}_{ \ell-1}^{\top}\|^{2}$ $\displaystyle=\operatorname{tr}\left((\bm{W}_{\ell}^{(a)}-\bm{P}_{\ell}\bm{W}_ {\ell}^{(b)}\bm{P}_{\ell-1}^{\top})(\bm{W}_{\ell}^{(a)}-\bm{P}_{\ell}\bm{W}_{ \ell}^{(b)}\bm{P}_{\ell-1}^{\top})^{\top}\right)$ (38) $\displaystyle=\operatorname{tr}\left(\bm{W}_{\ell}^{(a)}(\bm{W}_{\ell}^{(a)})^ {\top}\right)+\operatorname{tr}\left(\bm{W}_{\ell}^{(b)}(\bm{W}_{\ell}^{(b)})^ {\top}\right)$ (39) $\displaystyle\hskip 90.00014pt-2\operatorname{tr}\left(\bm{P}_{\ell}\bm{W}_{ \ell}^{(b)}\bm{P}_{\ell-1}^{\top}(\bm{W}_{\ell}^{(a)})^{\top}\right).$ (40) $$

We focus on the last term because only it depends on the permutation matrices. The SVDs of the weights $\bm{W}^{(a)}_{\ell}$ and $\bm{W}^{(b)}_{\ell}$ are denoted by $\bm{W}_{\ell}^{(a)}=\bm{U}_{\ell}^{(a)}\bm{S}_{\ell}^{(a)}(\bm{V}_{\ell}^{(a)}
)^{\top}=\sum_{i}\bm{u}_{\ell,i}^{(a)}s_{\ell,i}^{(a)}(\bm{v}_{\ell,i}^{(a)})^
{\top}$ and $\bm{W}_{\ell}^{(b)}=\bm{U}_{\ell}^{(b)}\bm{S}_{\ell}^{(b)}(\bm{V}_{\ell}^{(b)}
)^{\top}=\sum_{j}\bm{u}_{\ell,j}^{(b)}s_{\ell,j}^{(b)}(\bm{v}_{\ell,j}^{(b)})^
{\top}$, respectively. Thus, the last term of Equation 40 can be rewritten as

$$ $\displaystyle-2\operatorname{tr}\left(\bm{P}_{\ell}\bm{W}_{\ell}^{(b)}\bm{P}_{ \ell-1}^{\top}(\bm{W}_{\ell}^{(a)})^{\top}\right)=-2\sum_{i,j}s_{\ell,i}^{(a)} s_{\ell,j}^{(b)}(\bm{u}_{\ell,i}^{(a)})^{\top}(\bm{P}_{\ell}\bm{u}_{\ell,j}^{( b)})(\bm{v}_{\ell,i}^{(a)})^{\top}(\bm{P}_{\ell-1}\bm{v}_{\ell,j}^{(b)}).$ (41) $$

Therefore, Equation 2 equals

$$ $\displaystyle\operatorname*{arg\,min}_{\pi}\|\bm{\theta}_{a}-\pi(\bm{\theta}_{ b})\|^{2}=\operatorname*{arg\,max}_{\pi=(\bm{P}_{\ell})_{\ell}}\sum_{\ell,i,j} s_{\ell,i}^{(a)}s_{\ell,j}^{(b)}(\bm{u}_{\ell,i}^{(a)})^{\top}(\bm{P}_{\ell} \bm{u}_{\ell,j}^{(b)})(\bm{v}_{\ell,i}^{(a)})^{\top}(\bm{P}_{\ell-1}\bm{v}_{ \ell,j}^{(b)}),$ (42) $$

which completes the proof.
∎

###### Proof.

### G.3 Proof of Theorem F.1

###### Proof.

Note that the matrix $\bm{L}$ can be decomposed to $\bm{L}=\sum_{w}\bm{G}_{:,:,w}\otimes(\bm{e}_{w}\bm{e}_{w}^{\top})$ by using the tensor $\bm{G}$, where $\bm{e}_{w}$ is the orthonormal basis in Eucrlidean space $\bm{R}^{n^{2}}$. Thus, the SVD of $\bm{L}$ is given by

$$ $\displaystyle\bm{L}$ $\displaystyle=\sum_{w}\left(\sum_{i}\bm{u}_{w,i}s_{w,i}\bm{v}_{w,i}^{*}\right) \otimes(\bm{e}_{w}\bm{e}_{w}^{\top})$ (43) $\displaystyle=\sum_{w}\sum_{i}s_{w,i}(\bm{u}_{w,i}\bm{v}_{w,i}^{*})\otimes(\bm {e}_{w}\bm{e}_{w}^{\top})$ (44) $\displaystyle=\sum_{w}\sum_{i}s_{w,i}(\bm{u}_{w,i}\otimes\bm{e}_{w})(\bm{v}_{w ,i}\otimes\bm{e}_{w})^{*}.$ (45) $$

Thus, the SVD of $\bm{M}$ is also given by

$$ $\displaystyle\bm{M}$ $\displaystyle=\sum_{w}\sum_{i}s_{w,i}(\bm{I}_{m}\otimes\bm{Q})^{*}(\bm{u}_{w,i }\otimes\bm{e}_{w})(\bm{v}_{w,i}\otimes\bm{e}_{w})^{*}(\bm{I}_{m}\otimes\bm{Q})$ (46) $\displaystyle=\sum_{w}\sum_{i}s_{w,i}(\bm{u}_{w,i}\otimes\bm{Q}^{*}\bm{e}_{w}) (\bm{v}_{w,i}^{*}\otimes\bm{e}_{w}^{\top}\bm{Q}),$ (47) $$

which completes the proof.
∎

###### Proof.

### G.4 Proof of Theorem F.2

Before proving Theorem F.2, we first prove the following lemma:

###### Lemma G.1 .

Let $\bm{K}$ and $\bm{K}^{\prime}$ be kernels of convolutional layers. We have $\|\bm{M}-\bm{M}^{\prime}\|^{2}=n^{2}\|\bm{K}-\bm{K}^{\prime}\|^{2}$, where $\bm{M}$ and $\bm{M}^{\prime}$ are the matrix representations of $\bm{K}$ and $\bm{K}^{\prime}$, respectively.

###### Proof.

From the definition of $\bm{M}$ and $\bm{M}^{\prime}$, we have

$$ $\displaystyle\bm{M}=\begin{pmatrix}\bm{B}_{1,1}&\bm{B}_{1,2}&\dots&\bm{B}_{1,m }\\ \bm{B}_{2,1}&\bm{B}_{2,2}&\dots&\bm{B}_{2,m}\\ \vdots&\vdots&\ddots&\vdots\\ \bm{B}_{m,1}&\bm{B}_{m,2}&\dots&\bm{B}_{m,m}\end{pmatrix},~{}~{}\bm{M}^{\prime }=\begin{pmatrix}\bm{B}^{\prime}_{1,1}&\bm{B}^{\prime}_{1,2}&\dots&\bm{B}^{ \prime}_{1,m}\\ \bm{B}^{\prime}_{2,1}&\bm{B}^{\prime}_{2,2}&\dots&\bm{B}^{\prime}_{2,m}\\ \vdots&\vdots&\ddots&\vdots\\ \bm{B}^{\prime}_{m,1}&\bm{B}^{\prime}_{m,2}&\dots&\bm{B}^{\prime}_{m,m}\end{ pmatrix},$ (48) $$

where for any $c,d\in[m]$, $\bm{B}_{c,d}$ and $\bm{B}^{\prime}_{c,d}$ denote the doubly circulant matrices obtained from the kernels $\bm{K}$ and $\bm{K}^{\prime}$. Thus, $\|\bm{M}-\bm{M}^{\prime}\|^{2}=\sum_{c,d}\|\bm{B}_{c,d}-\bm{B}^{\prime}_{c,d}
\|^{2}=n\sum_{c,d}\sum_{i}\|\mathrm{circ}(K_{i,:,c,d})-\mathrm{circ}(K^{\prime
}_{i,:,c,d})\|^{2}=n^{2}\sum_{c,d}\sum_{i,j}(K_{i,j,c,d}-K^{\prime}_{i,j,c,d})
^{2}=n^{2}\|\bm{K}-\bm{K}^{\prime}\|^{2}$ holds.
∎

###### Proof of Theorem F.2 .

In convolutional layers, permutation matrices permute the input and output channels of the kernel. Therefore, the permutation matrices $\bm{P}_{\ell}$ and $\bm{P}_{\ell-1}$ corresponding to the input and output are $m\times m$ matrices. By using these matrices, the permutation of the matrix representation of the convolutional layer of the model $\bm{\theta}_{b}$ is denoted by $(\bm{P}_{\ell}\otimes\bm{I}_{m})\bm{M}^{(b)}(\bm{P}_{\ell-1}\otimes\bm{I}_{m})
^{\top}$. Lemma G.1 indicates that finding the permutation matrices that minimize the $L_{2}$ distance between the two kernels is equivalent to minimizing $\|\bm{M}^{(a)}-(\bm{P}_{\ell}\otimes\bm{I}_{m})\bm{M}^{(b)}(\bm{P}_{\ell-1}
\otimes\bm{I}_{m})^{\top}\|$. Therefore, we have

$$ $\displaystyle\|\bm{M}^{(a)}-(\bm{P}_{\ell}\otimes\bm{I}_{m})\bm{M}^{(b)}(\bm{P }_{\ell-1}\otimes\bm{I}_{m})^{\top}\|^{2}$ (49) $\displaystyle=\operatorname{tr}\left(\left(\bm{M}^{(a)}-(\bm{P}_{\ell}\otimes \bm{I}_{m})\bm{M}^{(b)}(\bm{P}_{\ell-1}\otimes\bm{I}_{m})^{\top}\right)\left( \bm{M}^{(a)}-(\bm{P}_{\ell}\otimes\bm{I}_{m})\bm{M}^{(b)}(\bm{P}_{\ell-1} \otimes\bm{I}_{m})^{\top}\right)^{*}\right)$ (50) $\displaystyle=\operatorname{tr}\left(\bm{M}^{(a)}(\bm{M}^{(a)})^{\top}\right)+ \operatorname{tr}\left(\bm{M}^{(b)}(\bm{M}^{(b)})^{\top}\right)-\operatorname{ tr}\left(\bm{M}^{(a)}\left((\bm{P}_{\ell}\otimes\bm{I}_{m})\bm{M}^{(b)}(\bm{P} _{\ell-1}\otimes\bm{I}_{m})^{\top}\right)^{*}\right)$ (51) $\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}-\operatorname{tr}\left((\bm{P}_{\ell}\otimes\bm{I}_{m})\bm{M}^{(b)}( \bm{P}_{\ell-1}\otimes\bm{I}_{m})^{\top}(\bm{M}^{(a)})^{*}\right).$ (52) $$

In Equation 52, the permutation matrices $\bm{P}_{\ell}$ and $\bm{P}_{\ell-1}$ are only related to the last two terms. Therefore, we focus only on them. By using some properties of trace, we have

$$ $\displaystyle-\operatorname{tr}\left(\bm{M}^{(a)}\left((\bm{P}_{\ell}\otimes \bm{I}_{m})\bm{M}^{(b)}(\bm{P}_{\ell-1}\otimes\bm{I}_{m})^{\top}\right)^{*} \right)-\operatorname{tr}\left((\bm{P}_{\ell}\otimes\bm{I}_{m})\bm{M}^{(b)}( \bm{P}_{\ell-1}\otimes\bm{I}_{m})^{\top}(\bm{M}^{(a)})^{*}\right)$ (53) $\displaystyle=-\operatorname{tr}\left(\bm{M}^{(a)}\left((\bm{P}_{\ell}\otimes \bm{I}_{m})\bm{M}^{(b)}(\bm{P}_{\ell-1}\otimes\bm{I}_{m})^{\top}\right)^{*} \right)-\operatorname{tr}\left(\left(\bm{M}^{(a)}\left((\bm{P}_{\ell}\otimes \bm{I}_{m})\bm{M}^{(b)}(\bm{P}_{\ell-1}\otimes\bm{I}_{m})^{\top}\right)^{*} \right)^{*}\right)$ (54) $\displaystyle=-\operatorname{tr}\left(\bm{M}^{(a)}\left((\bm{P}_{\ell}\otimes \bm{I}_{m})\bm{M}^{(b)}(\bm{P}_{\ell-1}\otimes\bm{I}_{m})^{\top}\right)^{*} \right)-\overline{\operatorname{tr}\left(\bm{M}^{(a)}\left((\bm{P}_{\ell} \otimes\bm{I}_{m})\bm{M}^{(b)}(\bm{P}_{\ell-1}\otimes\bm{I}_{m})^{\top}\right) ^{*}\right)}$ (55) $\displaystyle=-2\Re\operatorname{tr}\left(\bm{M}^{(a)}\left((\bm{P}_{\ell} \otimes\bm{I}_{m})\bm{M}^{(b)}(\bm{P}_{\ell-1}\otimes\bm{I}_{m})^{\top}\right) ^{*}\right).$ (56) $$

Here, from Theorem F.1,

$$ $\displaystyle\bm{M}^{(a)}$ $\displaystyle=\sum_{w,i}(\bm{u}_{w,i}^{(a)}\otimes\bm{Q}^{*}\bm{e}_{w})s_{w,i} ^{(a)}(\bm{v}_{w,i}^{(a)}\otimes\bm{Q}^{*}\bm{e}_{w})^{*}$ (57) $\displaystyle=\sum_{w,i}s_{w,i}^{(a)}(\bm{u}_{w,i}^{(a)}(\bm{v}_{w,i}^{(a)})^{ *}\otimes\bm{Q}^{*}\bm{e}_{w}\bm{e}_{w}^{\top}\bm{Q})$ (58) $\displaystyle=\sum_{w}(\bm{C}_{w}^{(a)}\otimes\bm{Q}^{*}\bm{e}_{w}\bm{e}_{w}^{ \top}\bm{Q}),$ (59) $$

and

$$ $\displaystyle(\bm{P}_{\ell}\otimes\bm{I}_{m})\bm{M}^{(b)}(\bm{P}_{\ell-1} \otimes\bm{I}_{m})^{\top}$ $\displaystyle=\sum_{w,i}s_{w,i}^{(b)}(\bm{P}_{\ell}\otimes\bm{I}_{m})(\bm{u}_{ w,i}^{(b)}\otimes\bm{Q}^{*}\bm{e}_{w})(\bm{v}_{w,i}^{(b)}\otimes\bm{Q}^{*}\bm{ e}_{w})^{*}(\bm{P}_{\ell-1}\otimes\bm{I}_{m})^{\top}$ (60) $\displaystyle=\sum_{w,i}s_{w,i}^{(b)}(\bm{P}_{\ell}\bm{u}_{w,i}^{(b)}(\bm{P}_{ \ell-1}\bm{v}_{w,i}^{(b)})^{*}\otimes\bm{Q}^{*}\bm{e}_{w}\bm{e}_{w}^{\top}\bm{ Q})$ (61) $\displaystyle=\sum_{w}(\bm{C}_{w}^{(b)}\otimes\bm{Q}^{*}\bm{e}_{w}\bm{e}_{w}^{ \top}\bm{Q})$ (62) $$

holds, where we let $\bm{C}_{w}^{(a)}=\sum_{i}s_{w,i}^{(a)}\bm{u}_{w,i}^{(a)}(\bm{v}_{w,i}^{(a)})^{*}$ and $\bm{C}_{w}^{(b)}=\sum_{i}s_{w,i}^{(b)}(\bm{P}_{\ell}\bm{u}_{w,i}^{(a)})(\bm{P}
_{\ell-1}\bm{v}_{w,i}^{(a)})^{*}$. From Equation 59 and Equation 62, we have

$$ $\displaystyle-2\Re\operatorname{tr}\left(\bm{M}^{(a)}\left((\bm{P}_{\ell} \otimes\bm{I}_{m})\bm{M}^{(b)}(\bm{P}_{\ell-1}\otimes\bm{I}_{m})^{\top}\right) ^{*}\right)$ (63) $\displaystyle=-2\Re\operatorname{tr}\left(\sum_{w}(\bm{C}_{w}^{(a)}\otimes\bm{ Q}^{*}\bm{e}_{w}\bm{e}_{w}^{\top}\bm{Q})\left(\sum_{w^{\prime}}(\bm{C}_{w^{ \prime}}^{(b)}\otimes\bm{Q}^{*}\bm{e}_{w^{\prime}}\bm{e}_{w^{\prime}}^{\top} \bm{Q})\right)^{*}\right)$ (64) $\displaystyle=-2\Re\operatorname{tr}\left(\sum_{w,w^{\prime}}\left(\bm{C}_{w}^ {(a)}(\bm{C}_{w^{\prime}}^{(b)})^{*}\otimes\bm{Q}^{*}\bm{e}_{w}\bm{e}_{w}^{ \top}\bm{e}_{w^{\prime}}\bm{e}_{w^{\prime}}^{\top}\bm{Q}\right)\right).$ (65) $$

Using the fact that if $w\neq w^{\prime}$, then $\bm{e}_{w}^{\top}\bm{e}_{w^{\prime}}=0$, and otherwise, $\bm{e}_{w}^{\top}\bm{e}_{w^{\prime}}=1$, we have

$$ $\displaystyle-2\Re\operatorname{tr}\left(\bm{M}^{(a)}\left((\bm{P}_{\ell} \otimes\bm{I}_{m})\bm{M}^{(b)}(\bm{P}_{\ell-1}\otimes\bm{I}_{m})^{\top}\right) ^{\top}\right)$ (66) $\displaystyle=-2\Re\sum_{w}\operatorname{tr}\left(\bm{C}_{w}^{(a)}(\bm{C}_{w}^ {(b)})^{*}\otimes\bm{Q}^{*}\bm{e}_{w}\bm{e}_{w}^{\top}\bm{Q}\right)$ (67) $\displaystyle=-2\Re\sum_{w}\operatorname{tr}\left(\bm{C}_{w}^{(a)}(\bm{C}_{w}^ {(b)})^{*}\right)\operatorname{tr}\left(\bm{Q}^{*}\bm{e}_{w}\bm{e}_{w}^{\top} \bm{Q}\right)$ (68) $\displaystyle=-2\Re\sum_{w}\operatorname{tr}\left(\bm{C}_{w}^{(a)}(\bm{C}_{w}^ {(b)})^{*}\right)$ (69) $\displaystyle=-2\Re\sum_{w}\operatorname{tr}\left(\left(\sum_{i}s_{w,i}^{(a)} \bm{u}_{w,i}^{(a)}(\bm{v}_{w,i}^{(a)})^{*}\right)\left(\sum_{j}s_{w,j}^{(b)}( \bm{P}_{\ell}\bm{u}_{w,j}^{(b)})(\bm{P}_{\ell-1}\bm{v}_{w,j}^{(b)})^{*}\right) ^{*}\right)$ (70) $\displaystyle=-2\Re\sum_{w}\sum_{i,j}s_{w,i}^{(a)}s_{w,j}^{(b)}\left((\bm{u}_{ w,i}^{(a)})^{*}(\bm{P}_{\ell}\bm{u}_{w,j}^{(b)})\right)^{*}\left((\bm{v}_{w,i} ^{(a)})^{*}(\bm{P}_{\ell-1}\bm{v}_{w,j}^{(b)})\right).$ (71) $$

From the above, the minimization of $\|\bm{M}^{(a)}-(\bm{P}_{\ell}\otimes\bm{I}_{m})\bm{M}^{(b)}(\bm{P}_{\ell-1}
\otimes\bm{I}_{m})^{\top}\|$ is equivalent to the maximization of $\Re\sum_{w}\sum_{i,j}s_{w,i}^{(a)}s_{w,j}^{(b)}\left((\bm{u}_{w,i}^{(a)})^{*}(
\bm{P}_{\ell}\bm{u}_{w,j}^{(b)})\right)^{*}((\bm{v}_{w,i}^{(a)})^{*}(\bm{P}_{
\ell-1}\bm{v}_{w,j}^{(b)}))$.
∎

###### Lemma G.1 .

###### Proof.

###### Proof of Theorem F.2 .

### G.5 Proof of | R ⁢ ( 𝜽 a , π ⁢ ( 𝜽 b ) ) | ≤ 1 𝑅 subscript 𝜽 𝑎 𝜋 subscript 𝜽 𝑏 1 |R(\bm{\theta}_{a},\pi(\bm{\theta}_{b}))|\leq 1 | italic_R ( bold_italic_θ start_POSTSUBSCRIPT italic_a end_POSTSUBSCRIPT , italic_π ( bold_italic_θ start_POSTSUBSCRIPT italic_b end_POSTSUBSCRIPT ) ) | ≤ 1

This subsection proves the following theorem:

###### Theorem G.2 .

Let $\bm{\theta}_{a}$ and $\bm{\theta}_{b}$ be the parameters of two MLPs with $L$-layers. For any permutation $\pi=(\bm{P}_{\ell})_{\ell\in[L]}$, we have

$$ $\displaystyle\left|R(\bm{\theta}_{a},\pi(\bm{\theta}_{b}))\right|=\frac{\left| \sum_{\ell,i,j}(\bm{u}_{\ell,i}^{(a)})^{\top}(\bm{P}_{\ell}\bm{u}_{\ell,j}^{(b )})(\bm{v}_{\ell,i}^{(a)})^{\top}(\bm{P}_{\ell-1}\bm{v}_{\ell,j}^{(b)})\right| }{\sum_{\ell}n_{\ell}}\leq 1.$ (72) $$

The equality holds if, for all $\ell,i$, $\bm{u}_{\ell,i}^{(a)}=\bm{P}_{\ell}\bm{u}_{\ell,i}^{(b)}$ and $\bm{v}_{\ell,i}^{(a)}=\bm{P}_{\ell-1}\bm{v}_{\ell,i}^{(b)}$.

###### Proof.

If, for all $\ell,i$, $\bm{u}_{\ell,i}^{(a)}=\bm{P}_{\ell}\bm{u}_{\ell,i}^{(b)}$ and $\bm{v}_{\ell,i}^{(a)}=\bm{P}_{\ell-1}\bm{v}_{\ell,i}^{(b)}$, then the equality obviously holds. Thus, we prove that Equation 72 holds. Using the property of trace, we have

$$ $\displaystyle\sum_{\ell,i,j}(\bm{u}_{\ell,i}^{(a)})^{\top}(\bm{P}_{\ell}\bm{u} _{\ell,j}^{(b)})(\bm{v}_{\ell,i}^{(a)})^{\top}(\bm{P}_{\ell-1}\bm{v}_{\ell,j}^ {(b)})$ $\displaystyle=\sum_{\ell,i,j}(\bm{P}_{\ell}\bm{u}_{\ell,j}^{(b)})^{\top}(\bm{u }_{\ell,i}^{(a)})(\bm{v}_{\ell,i}^{(a)})^{\top}(\bm{P}_{\ell-1}\bm{v}_{\ell,j} ^{(b)})$ (73) $\displaystyle=\sum_{\ell}\operatorname{tr}\left(\sum_{i}(\bm{u}_{\ell,i}^{(a)} )(\bm{v}_{\ell,i}^{(a)})^{\top}\sum_{j}\left((\bm{P}_{\ell}\bm{u}_{\ell,j}^{(b )})(\bm{P}_{\ell-1}\bm{v}_{\ell,j}^{(b)})^{\top}\right)^{\top}\right).$ (74) $$

Let $\bm{W}^{\prime{(a)}}_{\ell}=\sum_{i}(\bm{u}_{\ell,i}^{(a)})(\bm{v}_{\ell,i}^{(
a)})^{\top}$ and $\bm{W}^{\prime{(b)}}_{\ell}=\sum_{j}(\bm{P}_{\ell}\bm{u}_{\ell,j}^{(b)})(\bm{P
}_{\ell-1}\bm{v}_{\ell,j}^{(b)})^{\top}$. Obviously, they are matrices with all singular values of 1, and thus by using von Neuman’s trace inequality (von Neumann, ), we have

$$ $\displaystyle\sum_{\ell}\left|\operatorname{tr}\left(\bm{W}^{\prime{(b)}}_{ \ell}(\bm{W}^{\prime{(a)}}_{\ell})^{\top}\right)\right|\leq\sum_{\ell}n_{\ell}.$ (75) $$

Therefore, the triangle inequality yields that

$$ $\displaystyle\left|\sum_{\ell,i,j}(\bm{u}_{\ell,i}^{(a)})^{\top}(\bm{P}_{\ell} \bm{u}_{\ell,j}^{(b)})(\bm{v}_{\ell,i}^{(a)})^{\top}(\bm{P}_{\ell-1}\bm{v}_{ \ell,j}^{(b)})\right|$ $\displaystyle=\left|\sum_{\ell}\operatorname{tr}\left(\bm{W}^{\prime{(b)}}_{ \ell}(\bm{W}^{\prime{(a)}}_{\ell})^{\top}\right)\right|$ (76) $\displaystyle\leq\sum_{\ell}\left|\operatorname{tr}\left(\bm{W}^{\prime{(b)}}_ {\ell}(\bm{W}^{\prime{(a)}}_{\ell})^{\top}\right)\right|$ (77) $\displaystyle\leq\sum_{\ell}n_{\ell},$ (78) $$

which completes the proof.
∎

###### Theorem G.2 .

###### Proof.

### G.6 Proof of Theorem 4.2

###### Proof.

Because $\sigma$ is Lipschitz continuous with the constant $C$, we have

$$ $\displaystyle\mathbb{E}\|\sigma(\bm{W}_{\ell}^{(a)}\bm{z})-\sigma(\bm{W}_{\ell }^{(b)}\bm{z})\|\leq C\mathbb{E}\|\bm{W}_{\ell}^{(a)}\bm{z}-\bm{W}_{\ell}^{(b) }\bm{z}\|\leq C\sqrt{\mathbb{E}\|\bm{W}_{\ell}^{(a)}\bm{z}-\bm{W}_{\ell}^{(b)} \bm{z}\|^{2}},$ (79) $$

where we use Jensen’s inequality since the squre root function is concave. Focusing on the difference between the outputs in the square root, we get

$$ $\displaystyle\|\bm{W}_{\ell}^{(a)}\bm{z}-\bm{W}_{\ell}^{(b)}\bm{z}\|^{2}=\bm{z }^{\top}(\bm{W}_{\ell}^{(a)})^{\top}\bm{W}_{\ell}^{(a)}\bm{z}+\bm{z}^{\top}( \bm{W}_{\ell}^{(b)})^{\top}\bm{W}_{\ell}^{(b)}\bm{z}-2\bm{z}^{\top}(\bm{W}_{ \ell}^{(a)})^{\top}\bm{W}_{\ell}^{(b)}\bm{z}.$ (80) $$

From the SVDs of weights $\bm{W}_{\ell}^{(a)}=\sum_{i}\bm{u}_{\ell,i}^{(a)}s_{i}^{(a)}\bm{v}_{\ell,i}^{(
a)}$ and $\bm{W}_{\ell}^{(b)}=\sum_{i}\bm{u}_{\ell,i}^{(b)}s_{i}^{(b)}\bm{v}_{\ell,i}^{(
b)}$, we have $\bm{z}^{\top}(\bm{W}_{\ell}^{(a)})^{\top}\bm{W}_{\ell}^{(a)}\bm{z}=\sum_{i}(s_
{i}^{(a)})^{2}(\bm{v}_{\ell,i}^{(a)}\bm{z})^{2}$, $\bm{z}^{\top}(\bm{W}_{\ell}^{(b)})^{\top}\bm{W}_{\ell}^{(b)}\bm{z}=\sum_{i}(s_
{i}^{(b)})^{2}(\bm{v}_{\ell,i}^{(b)}\bm{z})^{2}$, and $\bm{z}^{\top}(\bm{W}_{\ell}^{(a)})^{\top}\bm{W}_{\ell}^{(b)}\bm{z}=\sum_{i,j}s
_{i}^{(a)}s_{j}^{(b)}(\bm{u}_{\ell,i}^{(a)})^{\top}\bm{u}_{\ell,j}^{(b)}(\bm{v
}_{\ell,i}^{(a)})^{\top}\bm{z}(\bm{v}_{\ell,j}^{(b)})^{\top}\bm{z}$. Therefore, Equation 79 can be rewritten as

$$ $\mathbb{E}\|\sigma(\bm{W}^{(a)}_{\ell}\bm{z})-\sigma(\bm{W}^{(b)}_{\ell}\bm{z} )\|\\ \leq C\sqrt{\sum_{i}(s_{\ell,i}^{(a)})^{2}\mathbb{E}((\bm{v}_{\ell,i}^{(a)})^{ \top}\bm{z})^{2}+\sum_{i}(s_{\ell,i}^{(b)})^{2}\mathbb{E}((\bm{v}_{\ell,i}^{(b )})^{\top}\bm{z})^{2}-2\sum_{i,j}s_{\ell,i}^{(a)}s_{\ell,j}^{(b)}(\bm{u}_{\ell ,i}^{(a)})^{\top}\bm{u}_{\ell,j}^{(b)}\mathbb{E}(\bm{v}_{\ell,i}^{(a)})^{\top} \bm{z}(\bm{v}_{\ell,j}^{(b)})^{\top}\bm{z}}.$ (81) $$

∎

###### Proof.

## Appendix H Additional Experimental results

### H.1 Distribution of Singular Values

Figure: (a) MLP, MNIST.
Refer to caption: x9.png

Figure 7 shows the distributions of the singular values of all the layers. The figure demonstrates that, in all layers except for the output layer, the singular values are very similar across all models. Meanwhile, in the output layer, there is variability in the singular values. However, this variability does not affect the accuracy of the merged model. Let $\bm{W}_{L}^{(a)}=\sum_{i}s^{(a)}_{L,i}\bm{u}^{(a)}_{L,i}(\bm{v}^{(a)})_{L,i}^{\top}$ and $\bm{W}_{L}^{(b)}=\sum_{i}s^{(b)}_{L,i}\bm{u}^{(b)}_{L,i}(\bm{v}^{(b)}_{L,i})^{\top}$ represent the output layer weights of the two trained models. The figure shows that the difference between the singular values of the two models is approximately a constant multiple. In other words, there exists a constant $\alpha$ such that $s_{L,i}^{(a)}\approx\alpha s_{L,i}^{(b)}$ for all $i$. If the singular vectors of the two weights are equal (i.e., $\bm{v}^{(a)}_{L,i}=\bm{v}^{(b)}_{L,i}$ and $\bm{u}^{(a)}_{L,i}=\bm{u}^{(b)}_{L,i}$ for all $i$), then $\bm{W}_{L}^{(a)}\approx\alpha\bm{W}_{L}^{(b)}$ holds (indeed, as mentioned in Section 4.3, the permutation matrix aligns the directions of the singular vectors). Therefore, the weight of the output layer of the merged model at the ratio $\lambda\in[0,1]$ is given by $\lambda\bm{W}_{L}^{(a)}+(1-\lambda)\bm{W}_{L}^{(b)}\approx\lambda\bm{W}_{L}^{(
a)}+(1-\lambda)\alpha\bm{W}_{L}^{(a)}=(\lambda+(1-\lambda)\alpha)\bm{W}_{L}^{(
a)}$. Thus, we can consider that the weight and the activation function of the merged model are given by $\bm{W}_{L}^{(a)}$ and a softmax function with an inverse temperature of $1/(\lambda+(1-\lambda)\alpha)$, respectively. Since the inverse temperature does not affect the accuracy value, the difference in the singular values of the output layer would not matter in satisfying LMC, at least in terms of accuracy.

### H.2 Inner Products Between Right Singular Vectors of Hidden Layers And Their Input

Figure: (a) MLP, MNIST.
Refer to caption: x13.png

Figure 8 shows the average absolute values of inner products between the right singular vectors and the input in each layer of models trained by SGD. The figure demonstrates that, except for the input and output layers, the singular vectors with larger singular values generally have larger inner products with the inputs. Note that the results of the input layers do not affect the permutation search based on WM because their right singular vectors are not changed by the permutations.

### H.3 Relationship with Model Width

Previous studies have demonstrated that the width of the model architecture affects the ease of achieving LMC. In this subsection, we explain this phenomenon based on the following three facts: as the model width increases, (i) the proportion of dominant singular values decreases, (ii) the right singular vectors corresponding to these dominant singular values will have large inner product values with the inputs of the hidden layers, and (iii) the WM preferentially aligns the directions of singular vectors corresponding to these dominant singular values.

#### (i) Dependency of model width on singular values.

Figure: Figure 9: Distribution of all singular values normalized by the largest one in the model as the model width multiplier changes. The vertical axis represents the singular values divided by the maximum singular value of each model, and the horizontal axis represents the ratio among all singular values (e.g., the point at 0.5 on the horizontal axis represents the singular value in the middle of all values sorted in descending order).
Refer to caption: x17.png

As we mentioned, the proportion of relatively large singular values in all singular values decreases as the model width increases. To verify this, Figure 9 shows the distribution of the singular values of all layers of VGG11 and ResNet20 trained on CIFAR10. Figure 9 shows the results of different model widths (i.e., dimensionality). As can be seen, the proportion of relatively large singular values decreases as the model width increases. Thus, the proportion of singular vectors that need to be aligned in the model decreases as the width increases.

#### (ii) Dependency of model width on inner products of right singular vectors.

Figure: (a) Width multiplier is 0.125.
Refer to caption: x18.png

Figure: (a) Width multiplier is 1.
Refer to caption: x24.png

Figure: (e) Width multiplier is 16.
Refer to caption: x28.png

We also investigated the effect of model width on the inner products between the hidden layer inputs and the right singular vectors. Figures 10 and 11 show the values of these inner products for each layer as model width changes. Figures 10 and 11 show the distributions of inner products for VGG11 and ResNet20 models trained on CIFAR10, respectively. These figures demonstrate that as model width increases, the inner products between the right singular vectors with large singular values and the inputs also increase.

#### (iii) Singular-vector alignment.

Figure: (a) Evaluation results of $R(\bm{\theta}_{a},\pi(\bm{\theta}_{b}))$ for all the singular vectors.
Refer to caption: x29.png

We conducted an experiment to examine how well the directions of singular vectors are aligned as model width increases when applying permutations found by WM. The results are shown in Figure 12. The figures display the evaluation of $R(\bm{\theta}_{a},\pi(\bm{\theta}_{b}))$ for the trained models $\bm{\theta}_{a}$ and $\bm{\theta}_{b}$ by searching for permutations $\pi$. For comparison, the case where no permutations are applied (i.e., $\pi$ is an identity map) is also shown. Additionally, a threshold $\gamma$ was introduced to assess the alignment of singular vectors with large singular values.

First, focusing on the results in Figure 12 with $\gamma=0$, we observe that the value of $R$ decreases even when the width increases and WM is used. Conversely, Figure 12 shows that the directions of singular vectors with particularly large singular values are aligned by permutation as model width increases. This suggests that even with WM, it is difficult to perfectly align the directions of singular vectors between the two models. However, increasing the width decreases the fraction of singular vectors with large singular values, and thus, making it easier for WM to align the directions of these dominant singular vectors.

As shown in Figure 9, when the model is sufficiently wide, the proportion of large singular values is very small compared to the total number of singular values. Furthermore, Figures 10 and 11 demonstrate that the right singular vectors associated with these relatively large singular values have a large inner product with the hidden layer input. This means that the number of singular vectors that WM needs to align to achieve LMC is reduced when the model is wide enough, as discussed in Section 4.4. Indeed, Figure 12 suggests that WM preferentially aligns these significant singular vectors. Therefore, increasing the width is expected to make LMC more feasible.

### H.4 Dependency of Weight Decay and Learning Rate

Qu & Horvath () have observed that strengthening weight decay and increasing the learning rate make it easier for LMC to be established through WM. In this section, we will explain this observation from the perspective of singular values of weights.

In Section 4, we stated that the reason why WM can establish LMC is that the permutation found by WM aligns singular vectors with large singular values between two models. In other words, the smaller the proportion of large singular values in the weights of each layer of the models, the more likely LMC is to be established by WM. In fact, some previous studies (Galanti et al., ; Timor et al., ) have shown that increasing the weight decay and learning rate during model training can reduce the ranks of the weights in the trained model. Therefore, it is highly likely that the results observed by Qu & Horvath () were caused by the reduction in the ranks of the weights. In the following, we will experimentally confirm this prediction.

Figure: (a) Evaluation results of ResNet20 ($\times 16$).
Refer to caption: x31.png

Figure 13 shows the experimental results of ResNet20 and VGG11 models. During model training, the learning rate was varied from ${0.0001,0.0003,\dots,0.03}$ and the weight decay strength was varied from ${0.00001,0.00003,\dots,0.003}$. For each condition, six models were trained, and model merging was performed three times by creating three pairs from them. The conditions for model training were the same as in Appendix D, except for the learning rate and weight decay. For VGG11, when the model width is quadrupled, the ratio of large singular values becomes very small regardless of the learning rate or weight decay, making it difficult to understand the relationship between the loss of the merged model and large singular values. Thus, the model width was doubled for VGG11 models. In addition, the permutation search method used in WM was based on the method of Ainsworth et al. ().

Figure 13 shows the test losses of the merged model and the original model, as well as the ratio of the number of large singular values to the width of the original model. Figure 13 displays the averaged results over three runs of model merging. The ratio in the figure was calculated as follows. Let $s_{\ell,1},s_{\ell,2},\dots,s_{\ell,n_{\ell}}$ be the singular values of the $\ell$-th layer, where $n_{\ell}$ represents the number of singular values, and $s_{\ell,1}$ is the largest singular value. Each singular value is divided by the largest singular value, and those whose ratio is 0.3 or more are counted (i.e., $s_{\ell,i}/s_{\ell,1}\geq 0.3$ for $i\in[n_{\ell}]$). Next, for all layers, we calculate the sum of these numbers and divide it by the sum of $n_{\ell}$ for all layers. This procedure can be written as $\sum_{\ell,i}I[s_{\ell,i}/s_{\ell,1}\geq 0.3]/\sum_{\ell}n_{\ell}$, where $I$ is an indicator function. In other words, this ratio becomes smaller when the number of singular values that are relatively large compared to the maximum singular value is small compared to the width of the model.

From Figure 13, we can see that the ratio of large singular values decreases as both the weight decay and the learning rate increase. This has already been noted in previous research (Galanti et al., ; Timor et al., ). Additionally, Figure 13 shows that the test loss of the merged model decreases when both the test loss of the original model and the ratio of large singular values are small. This can be explained by our analysis in Section 4. As we described, WM facilitates LMC by aligning the singular vectors between the two models and making the functions of the middle layers of the merged model and the original model more similar. In other words, this suggests that it is challenging for the merged model to outperform the original model. Furthermore, the smaller the ratio of large singular values, the easier it is to align the singular vectors using WM, so the functions of the merged model and the original model become more closely aligned. From this, we conclude that the higher the performance of the original model and the smaller the ratio of large singular values, the better the performance of the merged model. This is consistent with the results shown in Figure 13.

### H.5 Activation Matching

**Table 3: Model Merging results with AM and the estimated barrier value using Taylor approximation when $\lambda=1/2$**
| Dataset | Network | Test acc. | Barrier ($\lambda=1/2$) | Taylor approx. | $\|\bm{\theta}_{a}-\bm{\theta}_{b}\|$ | $\|\bm{\theta}_{a}-\pi(\bm{\theta}_{b})\|$ | Reduction rate [%] |
| --- | --- | --- | --- | --- | --- | --- | --- |
| CIFAR10 | VGG11 | $88.786\pm 0.186$ | $0.077\pm 0.044$ | $2.491\pm 0.266$ | $799.503\pm 16.396$ | $742.300\pm 18.526$ | $7.161\pm 0.572$ |
| ResNet20 | $89.190\pm 0.192$ | $0.189\pm 0.031$ | $7.431\pm 0.667$ | $710.762\pm 16.261$ | $671.373\pm 13.816$ | $5.538\pm 0.226$ |  |
| FMNIST | MLP | $88.356\pm 0.221$ | $-0.236\pm 0.024$ | $0.948\pm 0.173$ | $121.853\pm 5.830$ | $108.968\pm 5.365$ | $10.578\pm 0.343$ |
| MNIST | MLP | $98.274\pm 0.084$ | $-0.020\pm 0.004$ | $0.064\pm 0.035$ | $81.231\pm 5.580$ | $68.722\pm 5.197$ | $15.428\pm 1.037$ |

Figure: Figure 14: Evaluation results of $R(\bm{\theta}_{a},\bm{\theta}_{b})$ with and without AM.
Refer to caption: x33.png

Figure: (a) Evaluation results with the threshold $\gamma=0$.
Refer to caption: x34.png

Table 3 shows the results of model merging using AM under the same conditions as in Section 3. The table indicates that the loss barrier is sufficiently small when AM is applied. Interestingly, AM reduces the distance between the two models to the same extent as WM.

Figure 14 shows the value of $R$ between the two models $\bm{\theta}_{a}$ and $\bm{\theta}_{b}$. The figure clearly shows that the value of $R$ is larger using AM when $\gamma=0.3$. This indicates that AM aligns the directions of singular vectors with large singular values for the two models, similar to the result of WM (Figure 2). To further examine the relationship of singular vectors between the models before and after merging, Figure 15 shows the values of $R$ between these models. This result also demonstrates a similar trend to the WM results shown in Figure 3, suggesting that the reasons for the establishment of the LMC are almost the same for both WM and AM.

### H.6 STE and WM

**Table 4: Evaluation results of barrier between each pair of models**
|  |  |  | Loss barrier | Accuracy barrier |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Dataset | Network | $(\bm{\theta}_{a}+\pi_{b}(\bm{\theta}_{b}))/2$ | $(\bm{\theta}_{a}+\pi_{c}(\bm{\theta}_{c}))/2$ | $(\pi_{b}(\bm{\theta}_{b})+\pi_{c}(\bm{\theta}_{c}))/2$ | $(\bm{\theta}_{a}+\pi_{b}(\bm{\theta}_{b}))/2$ | $(\bm{\theta}_{a}+\pi_{c}(\bm{\theta}_{c}))/2$ | $(\pi_{b}(\bm{\theta}_{b})+\pi_{c}(\bm{\theta}_{c}))/2$ |
| WM | CIFAR10 | VGG11 | $0.094\pm 0.158$ | $0.037\pm 0.156$ | $0.141\pm 0.141$ | $8.362\pm 5.677$ | $7.555\pm 4.978$ | $10.12\pm 5.117$ |
| ResNet20 | $0.135\pm 0.026$ | $0.098\pm 0.011$ | $0.294\pm 0.098$ | $3.312\pm 0.61$ | $2.995\pm 0.064$ | $7.23\pm 0.99$ |  |  |
| FMNIST | MLP | $-0.211\pm 0.029$ | $-0.174\pm 0.044$ | $-0.174\pm 0.051$ | $1.947\pm 0.501$ | $1.703\pm 0.289$ | $4.337\pm 1.434$ |  |
| MNIST | MLP | $-0.027\pm 0.005$ | $-0.034\pm 0.003$ | $-0.031\pm 0.003$ | $0.173\pm 0.04$ | $0.198\pm 0.032$ | $0.475\pm 0.069$ |  |
| STE | CIFAR10 | VGG11 | $0.081\pm 0.031$ | $0.099\pm 0.042$ | $2.172\pm 0.989$ | $4.86\pm 0.815$ | $5.76\pm 0.537$ | $32.013\pm 8.193$ |
| ResNet20 | $0.466\pm 0.154$ | $0.446\pm 0.138$ | $1.693\pm 0.168$ | $15.005\pm 3.765$ | $13.942\pm 4.008$ | $34.483\pm 2.426$ |  |  |
| FMNIST | MLP | $-0.372\pm 0.016$ | $-0.343\pm 0.055$ | $0.023\pm 0.118$ | $2.667\pm 0.248$ | $2.483\pm 0.621$ | $15.97\pm 1.724$ |  |
| MNIST | MLP | $-0.037\pm 0.011$ | $-0.039\pm 0.006$ | $0.017\pm 0.014$ | $0.253\pm 0.176$ | $0.358\pm 0.198$ | $2.312\pm 0.457$ |  |

In this subsection, additional experimental results for Section 6.3 are shown in Table 4 for the barrier values and $R$ values between each pair of models. The table shows the model-merging results with $\lambda=1/2$, and the mean and standard deviation of five model merges. In the table, a negative value for the barrier indicates an improvement in performance due to the merging. From the Table 4, we can see that the barrier between $\pi_{b}(\bm{\theta}_{b})$ and $\pi_{c}(\bm{\theta}_{c})$ is also smaller for WM than for STE.

### H.7 Dependency of R 𝑅 R italic_R on Threshold γ 𝛾 \gamma italic_γ

Figure: (a) $R$ value between the models $\bm{\theta}_{a}$ and $\bm{\theta}_{b}$.
Refer to caption: x36.png

Figure 16 shows the value of $R$ when the threshold $\gamma$ is varied. Figure 16 displays the $R$ value between model $\bm{\theta}_{a}$ and the permuted model $\pi(\bm{\theta}_{b})$, along with the $R$ value before permutation for comparison. Figure 16 illustrates the $R$ values between the merged model $(\bm{\theta}_{a}+\pi(\bm{\theta}_{b}))/2$ and the original models $\bm{\theta}_{a}$ and $\bm{\theta}_{b}$, also including the $R$ value without permutation for comparison.

In Figure 16, the $R$ value between the original models is nearly zero regardless of the $\gamma$ value. However, when permutation is applied, the $R$ value increases as $\gamma$ increases. This indicates that WM preferentially aligns the directions of the larger singular vectors between models $\bm{\theta}_{a}$ and $\bm{\theta}_{b}$. As shown in Figure 16, this effect helps align the singular vectors with larger singular values between the merged and original models. Aligning these singular vectors more closely makes LMC more feasible because the outputs between the two models are closer, as discussed in Section 4.4.

### H.8 LMC on ResNet50 trained on ImageNet

**Table 5: Results of model merging of ResNet50 models trained on ImageNet dataset.**
|  | $\bm{\theta}_{c}$ w/ WM | $\bm{\theta}_{c}$ w/o WM | $\bm{\theta}_{a}$ | $\bm{\theta}_{b}$ |
| --- | --- | --- | --- | --- |
| Test loss | $5.207\pm 0.073$ | $6.897\pm 0.001$ | $1.491\pm 0.011$ | $1.493\pm 0.007$ |
| Test acc. | $40.239\pm 2.088$ | $0.179\pm 0.020$ | $75.741\pm 2.088$ | $75.856\pm 0.107$ |

Figure: Figure 17: Evaluation results of $R$ values between each pair of ResNet50 models trained on ImageNet dataset.
Refer to caption: x38.png

In the paper, most of the analysis was performed on relatively small datasets such as MNIST and CIFAR10. In this subsection, we train ResNet50 models on a larger dataset, ImageNet, and analyze the results of model merging based on WM.

#### Experimental results of model merging.

Figure: Figure 18: Distributions of the singular values of each layer of ResNet50 models trained on ImageNet dataset.
Refer to caption: x39.png

Figure: Figure 19: Average absolute values of inner products between the right singular vectors and the input of each layer in ResNet50 trained on ImageNet dataset ($\times 10^{6}$).
Refer to caption: x40.png

Table 5 presents the results of merging ResNet50 models trained on the ImageNet dataset. Table 5 shows the test loss and top-1 accuracy of the models before and after merging (i.e., the pre-merged models $\bm{\theta}_{a}$ and $\bm{\theta}_{b}$, and the merged model $\bm{\theta}_{c}$). Table 5 shows the $L_{2}$ distance between models $\bm{\theta}_{a}$ and $\bm{\theta}_{b}$ before and after applying permutations. These tables report the mean and standard deviation for five independent model merges. According to Table 5, the test loss and accuracy of the merged model are clearly improved by using WM. However, they are still worse than those of the pre-merged models $\bm{\theta}_{a}$ and $\bm{\theta}_{b}$, indicating that the LMC cannot be considered satisfied. Table 5 demonstrates that using WM decreases the $L_{2}$ distance between models $\bm{\theta}_{a}$ and $\bm{\theta}_{b}$. This decrease in $L_{2}$ distance is larger than that observed for VGG11 and ResNet20 in Table 1, suggesting that the singular vectors of models $\bm{\theta}_{a}$ and $\bm{\theta}_{b}$ are better aligned by permutations. Figure 17 presents the results of evaluating $R$ for each model pair. When $\gamma=0.3$, Figure 17 shows that $R(\bm{\theta}_{a},\bm{\theta}_{b})$ rises to about 0.5 with WM, and the value of $R$ increases to approximately 0.9 between the pre- and post-merged models (i.e., $R(\bm{\theta}_{a},\bm{\theta}_{c})$ and $R(\bm{\theta}_{b},\bm{\theta}_{c})$). Thus, even with ResNet50, the singular vectors with large singular values are aligned between the pre- and post-merged models by using WM.

To investigate why LMC does not hold even though the singular vectors with large singular values are aligned by using WM, we examine the distributions of singular values at each layer and the inner products between right singular vectors and the inputs. Figure 18 shows the distribution of singular values for each layer, and Figure 19 shows the distribution of the average absolute values of the inner products between the right singular vectors and the inputs for each layer. Each figure is plotted in a different color for the 10 trained models. Figure 18 demonstrates that the distributions of singular values are nearly identical across all models. Thus, the difference in singular values between models is not a reason for preventing LMC. In Figure 19, focusing on the distribution of the inner product between the right singular vectors and the inputs, we observe that the right singular vectors with large singular values does not necessarily have a large inner product with the input. As discussed in Section 4.4, WM can only align singular vectors with large singular values, so this discrepancy can be a reason preventing the establishment of LMC. As shown in Figures 11 and 10, the wider the model, the larger the inner product of the input and the right singular vectors with large singular values in the hidden layers. Therefore, it is considered necessary to increase the width of the model to establish LMC with ResNet50.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - What explains that permutations found by activation matching are much different from weight matching? Can we use SVD to analyze the qualitative / quantitative differences between the two methods that ostensibly minimize a similar objectives, i.e. matching the singular values.
- In [2], authors also analyzed linear connectivity of more than 3 models for checking strong linear connectivity where it is found that as width of the model increases, weight matching allows strong linear connectivity. Are there any insights we can gain from SVD when it studying strong linear connectivity.

[2] Sharma, E., Kwok D., Denton T., Roy DM, Rolnick D., and  Dziugaite G.K.: Simultaneous linear connectivity of neural networks modulo permutation. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (2024).
soundness: 3
strengths: - Analysis of the weight matching via SVD of layerwise weight matrices is interesting.
- I find the observation that weight alignment leads to linear connectivity of more that 2 models also interesting
summary: This paper analyses the the weight matching algorithm through singular value decomposition. 

Paper starts with the observation that algorithms like weight matching do not significantly reduce the  l2 distance. Followed by this observation, authors empirically verify that weight matching algorithm does not bring the models close enough such that the error barrier can be calculated by a second order Taylor approximation. 

Authors then made the observation that minimizing the l2 distance wrt permutations is equivalent to maximizing the inner product wrt permutations. Using this relationship authors show that weight matching indeed matches the singular values of the layer-wise weight matrices. 

Authors empirically validate that weight-matching and activation matching align the singular vectors however STE does not match the STE. 

Authors empirically show that aligning via SVD can lead to better alignment for 3 models.
weaknesses: - I would suggest authors to keep the vectorized notation instead of summations to make the paper an easier read.
- Observation that minimizing the l2 distance is equivalent to  maximizing the inner product is pretty well know. [1] also leverages this observation to implement the weight matching algorithm. I found those contributions incremental.

[1] Ainsworth, S., Hayase, J., Srinivasa, S.: Git re-basin: merging models modulo permutation symmetries. In: The Eleventh International Conference on Learning Representations (2023)

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: - Connecting it back to the idea in the intro about SGD and the loss landscape, do the authors have any thoughts on how their findings relate to the efficacy of SGD?
- How well does this analysis scale beyond 2 or 3 models? "Therefore, WM is likely to be more advantageous, especially for merging three or more models" (line 522), is there additional proof of this?
soundness: 3
strengths: - This work covers a very important topic likely to have a broad impact in understanding learning and combining in deep neural networks. 
- Paper is very well written; notation is clear and consistent, the problem is motivated, contributions are clear. 
- The Taylor approximation to estimate the barrier for merging models provides a clear intuition for their approach, and demonstrate that WM permutations may not result in the best merge. Results in Table 1 provide good evidence for this stance (though perhaps too many unnecessary significant figures are given and appear arbitrary as a result). 
- Results overall support the paper claims and demonstrate the different merging means for WM, STE, and AM.
summary: This work presents a novel analysis of weight matching (WM) for model merging, with comparisons to AM and STE. The authors demonstrate that permutations found by WM may not reduce the L2 distance between 2 models, albeit LMC is satisfied. Instead they show that the singular vectors are aligned. They also demonstrate that STE merging does not align singular vectors.
weaknesses: - Table 1 (and others) could be improved through highlighting sections to support the conclusion, e.g., where some solutions violate the nearness assumption and the Taylor approx. is not good. Many values presented are overlapping with over values and it is not immediately clear which are significant different vs not. 
- Figures are small and difficult to read. For the distribution plots (Fig. 1, 4), perhaps a logscale on the y axis would allow the reader to better evaluate the claims. 
- "the barrier between πb(θb) and πc(θc) is smaller with WM than with STE. This means that there is a significant difference between the principles of permutations obtained by WM and STE." (line 519) is difficult to evaluate by eye in Figure 6 and not particularly convincing. 
- The statement "suggesting that the reason AM achieves LMC is likely the same as for WM" (line 458) seems overly confident given the evidence presented. Are there other explanations for the similar results?

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: * For the sentence in line 139 "Entezari et al. (2022) conjectured that for SGD solutions ...": is this still a conjecture, or has a more solid theoretical foundation been established?

* In line 122, why is $W_{\ell + 1}' = W_{\ell + 1} P^{T}$ while $W_\ell' = P W_\ell$? I thought it should be $W_{\ell + 1}' = P W_{l+1} $
soundness: 3
strengths: * The paper is clearly written and easy to follow. 

* I have checked the proofs of the theorems presented in the main paper, and they appear correct and clearly articulated.

* The observation that $L_2$ distance reduction via WM is not the sole cause of LMC is interesting. This observation is supported with numerical experiments.

* The theoretical analysis is clearly explained when the underlying assumptions hold (though I have questions about the assumptions' validity; see the Weaknesses section).

* The appendix provides a good amount of detail for reproducing experiments, although the code itself is not shared.
summary: This paper analyses the linear mode connectivity (LMC) using weight matching (WM), activation matching (AM), and straight-through estimator (STE), with a primary focus on WM. The authors propose that, even though WM does not significantly decrease the $L_2$ distance of the weights between two independently trained models, it still establishes LMC by aligning the singular vectors of each layer associated with the largest singular values. Empirically, the paper demonstrates that two models with a large $L_2$ weight distance after WM can nonetheless produce similar outputs, and empirical results support that singular vectors with largest singular values across the layers are aligned with WM process. Furthermore, it has been demonstrated that AM process exhibits a similar effect, while STE behaves differently. The theoretical analysis leverages SVD on layer weights, and numerical experiments using VGG, ResNet, and MLP on MNIST, CIFAR10, and ImageNet support the claims.
weaknesses: * In line 70, the sentence "... this intuition is inaccurate ..." raises a concern. I think the intuition presented above this sentence is correct, i.e., if $\theta_a \approx \pi (\theta_b)$ then it is reasonable to think that the interpolated model will produce similar outputs. I believe the main narrative of the paper appears to be that even when $\theta_a \not\approx \pi (\theta_b)$, where $\pi$ is found by WM, the interpolated model can produce approximately equal outputs. 

* In line 211, the phrase "... algorithm similar to the backpropagation method to efficiently compute $\mu^T H$ ..." would benefit from further clarification in the appendix. I couldn’t locate a section elaborating on this point. What specifically do the authors mean by "similar algorithm to backprop"?

* Regarding the sentence in line 261 "... we can find permutation matrices such that the left and right singular vectors of the two models match ...": it seems unlikely that all singular vectors can be matched purely through permutation in all scenarios.

* In line 286 "distributions of the singular values of the weights ... almost equal": This seems like a strong claim to me. The paper presents results with common datasets like MNIST and CIFAR10 using MLP, VGG, and ResNet. I think the conclusion requires further evidence or a more theoretical foundation. The appendix H.1 discusses this in a more detailed manner, and Figure 7 indicates that output layers' singular value distribution have variability. For this case, the paper suggests that the singular values are approximately a constant multiple across the two models. Even though the output layers' singular value distributions seem correlated, the "constant multiple" assumption seems like a strong one to me as there are contradicting examples in Figure 7 (e.g. Figure 7(a) light blue curve and red curve).

* The assumption that the input vector's direction aligns with the singular vectors associated with large singular values also seems like a strong claim. Analysis in Appendix H.8 with ResNet50 on ImageNet shows that this assumption does not always hold.

* The abbreviation "layer-wise model compression (LMC)" in line 890 conflicts with the abbreviation for linear mode connectivity, and I believe layer-wise model compression does not need an abbreviation.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 4
questions: 1 - Both WM and AM are iterative algorithms. How does the L_2 distance evolve with an increasing number of iterations? It is surprising that the distance shrinkage is nearly identical between WM and AM, given that AM corresponds to layer-wise feature matching, which is closely linked to reduced distances between models.

2 - While the experimental results indicate that the Taylor approximation is imprecise for estimating the barrier, could it be that the models are still relatively close? Since L_2 distance is difficult to interpret in high-dimensional spaces, do you think that including the third term of the Taylor decomposition might help, or that a more precise analysis of the barrier (not just at the midpoint) could yield better insights?

3 - How does your analysis account for the necessity of post-matching fine-tuning, which is particularly critical for networks with batch normalization layers (e.g., Jordan et al., Repair: Renormalizing Permuted Activations for Interpolation Repair)?
soundness: 3
strengths: The study offers a valuable perspective on analyzing LMC, a phenomenon that characterizes the properties of loss surfaces and solutions discovered by stochastic gradient descent (SGD) and its analogs. The results are presented clearly and are substantiated both empirically and, in part, theoretically.
summary: This paper analyzes the algorithms for enforcing linear mode connectivity (LMC) as discussed in Ainsworth et al. (2023). The central question addressed is the underlying mechanisms that lead to the removal of barriers when applying weight matching (WM), activation matching (AM), or the straight-through estimator (STE). The authors argue that the primary reason WM and AM achieve linear connectivity between models is the alignment of the eigenvectors corresponding to large eigenvalues of the weights. Theoretically, they demonstrate that similar eigenvectors lead to similar outputs at the layer level, and that minimizing the L_2 distance between weights is equivalent to aligning these eigenvectors. Furthermore, the paper suggests that STE operates through a different mechanism to achieve low-barrier perturbations.
weaknesses: A key weakness is the lack of significant novelty in the paper's primary insights. Similar observations have been made in previous works, such as [1] and [2]. These papers also explore Hessian-based barrier approximations and spectral analysis to explain LMC. Although this does not undermine the findings of the current paper, these existing works should be cited and discussed.

Another concern is the transition to per-layer analysis in the discussion of WM. The authors claim that eigenvector alignment results in aligned representations, but this can be challenged. Firstly, the use of the Lipschitz constant in the analysis limits the result to essentially one activation only. When multiple layers are involved, the Lipschitz constant becomes excessively large. Secondly, [3] show that there are often no barriers at the layer level (even when barriers exist at the full network level). Although this could still be explained by eigenvectors, it weakens the connection between similar activations on the level of one layer and the lack of barriers at the full-model level.

The analysis of STE seems rather straightforward, given that this algorithm is designed solely to minimize loss at a single point (the midpoint on the linear path). Unlike WM or AM, it does not aim to identify fully "matching" models. The numerical results in Table 4 are also not fully convincing, as the difference between the barriers found using WM and STE is minimal.

Additionally, the paper claims that models trained in the same manner exhibit higher barriers at the midpoint between them. However, in the experiments of [3], it is shown that when one model generalizes better than another, the higher barrier shifts. Therefore, the assumptions in Section 3.2 require further clarification.

Minor Comments:

1 - In Section 4.4, please specify that the results in Appendix H3 and H4 are empirical.

2 - I suggest replacing Figure 6 with Table 4, as low-dimensional visualizations provide an imprecise representation of loss surface properties.

[1] Singh et al., Landscaping Linear Mode Connectivity

[2] Yunis et al., Approaching Deep Learning through the Spectral Dynamics of Weights

[3] Adilova et al., Layerwise Linear Mode Connectivity

Meta-review: The paper 'Analysis of Linear Mode Connectivity via Permutation-Based Weight Matching: With Insights into Other Permutation Search Methods' was reviewed by 4 reviewers who gave it an average score of 7.0 (final scores: 6+6+8+8). The reviewers found this work relevant for the conference and all of them recommend accepting this submission.

Decision: Accept (Poster)