Do We Really Need Permutations? Impact of Model Width on Linear Mode Connectivity
Authors: Akira Ito
Masanori Yamada
Daiki Chijiwa
Atsutoshi Kumagai
Venue: ICLR 2026 Poster (2026)
Keywords: deep learning
linear mode connectivity
permutation symmetries
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: Width expansion probably facilitates linear mode connectivity without permutations.

Abstract: Recently, Ainsworth et al. empirically demonstrated that, given two independently trained models, applying a parameter permutation that preserves the input–output behavior allows the two models to be connected by a low-loss linear path. When such a path exists, the models are said to achieve linear mode connectivity (LMC). Prior studies, including Ainsworth et al. (2023), have reported that achieving LMC requires not only an appropriate permutation search but also sufficiently wide models (e.g., a 32 $\times$ width multiplier for ResNet-20). This is broadly believed to be because increasing the model width ensures a large enough space of candidate permutations, increasing the chance of finding one that yields LMC. In this work, we empirically demonstrate that, __even without any permutations__, simply widening the models is sufficient for achieving LMC when using a suitable softmax temperature calibration. We further explain why this phenomenon arises by analyzing intermediate layer outputs. Specifically, we introduce layerwise exponentially weighted connectivity (LEWC), which states that the output of each layer of the merged model can be represented as an exponentially weighted sum of the outputs of the corresponding layers of the original models. Consequently the merged model's output matches that of an ensemble of the original models, facilitating LMC. To the best of our knowledge, this work is the first to show that widening the model not only facilitates __nonlinear__ mode connectivity, as suggested in prior research, but also significantly increases the possibility of achieving __linear__ mode connectivity.

Published as a conference paper at ICLR 2026
Do WE REALLY NEED PERMUTATIONS? IMPACT OF MODEL WIDTH ON LINEAR MODE CONNECTIVITY
Akira Itol Masanori Yamada Daiki Chijiwa² & Atsutoshi Kumagai 'Tohoku University "NTT DOCOMO, INC. "NTT Computer and Data Science Laboratories akira. ito. bl@tchoku. ac. jp. [masanori, yamada, daiki daiki.chi jiwa, atsutoshi. kumagai kuwagai/entt.coomm
ABSTRACT
Recently, Ainsworth et al. empirically demonstrated that, given two independently trained models, applying parameter permutation that preserves the input-output behavior allows the two models to be connected by low-loss linear path. When such path exists, the models are said to achieve linear mode connectivity (LMC). Prior studies, including Ainsworth al. (2023), have reported that achieving LMC requires not only an appropriate permutation search but also sufficiently wide models (e.g.. 32 width multiplier for ResNet-20). This is broadly believed to be because increasing the model width ensures a large enough space of candidate per- mutations, increasing the chance of finding one that yields L.C.. In this work, we empirically demonstrate that, even without any permutations, simply widening the models is sufficient for achieving LMC when using a suitable softmax temperature calibration. We further explain why this phenomenon arises by analyzing interme- diate layer outputs. Specifically, we introduce layerwise exponentially weighted connectivity (LEWC), which states that the output of each layer of the merged model can be represented as an exponentially weighted sum of the outputs of the corresponding layers of the original models. Consequently the merged model's output matches that of an ensemble of the original models, facilitating LMC. To the best of our knowledge, this work the first show that widening the model not only facilitates nonlinear mode connectivity, as suggested in prior research, but also significantly increases the possibility of achieving linear mode connectivity.
INTRODUCTION
Large neural networks (NNs) are widely used across various domains (Vaswani et al., 2017; van den Ot 2016; Zhao et al., 2023), and optimizing their parameters constitutes a massive non-convex optimization problem. Remarkably, stochastic gradient descent (SGD), which is widely employed for NN training, is known to find good solutions despite its simplicity. One hypothesis proposed to explain this seemingly counterintuitive phenomenon is that the landscape of the loss function may be far simpler than previously thought. Several studies (Garipov et al., 2018; Draxler et al., 2018; Freeman & Bruna, 2017) have reported that different NN solutions can be connected through simple nonlinear paths with almost increase in loss. Recently, Entezari et al. (2022) conjectured that, after accounting for all permutation symmetries in NNs, Conjecture 1.1 may hold.
Conjecture 1.1 (Permutation invariance, informal). Let O, and O, be the parameters of two models. When their model widths are sufficiently large, there exists permutation т such that the barrier between 0, and π (0,) (as defined in Definition 2.1) becomes sufficiently small with high probability.
Here, the barrier refers to the amount of loss increase when linearly interpolating between the weights of the two models. When the barrier between them is sufficiently small, they are said to exhibit linear mode connectivity (LMC) (Frankle et al., 2020). Conjecture 1.1 claims that many SGD solutions can be mapped into the same loss basin by applying an appropriate permutation. Indeed. several studies (Ainsworth et al., 2023; Singh & Jaggi, 2020) have experimentally validated this conjecture across a variety of datasets and models by employing effective permutation-finde findding techniques.
Work done while NTT Social Informaties I Lahoratories.

---

Published as a conference paper at ICLR 2026
In previous studies (Entezari et al., 2022; Ainsworth et al., 2023; Singh et al., 2024; Ito et al., 2025a;b; Ferbach et al., 2024), it has been widely believed that sufficiently widening the model is required to find permutation under which LMC holds. Intuitively, if the model is not sufficiently wide, the number of candidate permutations is limited, making it difficult to discover an appropriate permutation that brings the models into the same loss basin. For example, Ainsworth et al. (2023) empirically demonstrated that, without increasing ResNet-20's width by 32x or VGG-16's by 4x, permutations fail to sufficiently reduce the loss barrier on the CIFAR-I0 dataset. In addition, several studies have shown that widening ResNet-50 trained on the ImageNet dataset improves the test accuracy of the merged model when interpolating the weights of two trained models with permutations (Ainsworth et al., 2023). The original conjecture (Conjecture 1.1) also suggests that there may be no permutation that enables LMC unless the model is sufficiently wide.
In this paper, we empirically show that once the model is sufficiently wide, simply averaging the weights of independently trained models without applying any permutation achieves test accuracy comparable to that of the original models. This finding implies that even without aligning models to the same loss basin via permutation, sufficiently widening the models naturally places them within the same basin in terms of test accuracy. Prior works (Nguyen, 2019; Shevchenko & Mondelli, 2020) have demonstrated that increasing model width facilitates the existence of nonlinear low-loss paths between trained models. However, the best of our knowledge, no study has suggested that it also facilitates connectivity through linear paths. Our results suggest that widening the model itself may play more critical role in achieving LMC than increasing the number of possible permutations.
Understanding the principle behind LMC is important not only for theoretical reasons, such as explaining the effectiveness of SGD in deep learning, but also for practical applications like model merging. Prior work (Singh & Jaggi, 2020; Wang et al., 2020; Guerrero- et al., 2023) has leveraged permutation symmetries in neural networks for model merging, federated learning, and continual learning. While weight averaging is known to work well for models fine-tuned from a shared foundation model, merging independently trained models remains challenging. By studying LMC without permutations, we suggest that similar strategy may also be feasible in this setting.
Contributions
The contributions of this paper are threefold:
1. Widening improves the performance of merged models. We empirically show that just increasing the width of independently trained models monotonically improves the accuracy of their merged model without permutations, eventually matching the performance of the original models. Furthermore, we show that the loss barrier can be reduced to nearly zero by calibrating the softmax with an appropriate inverse temperature, thereby achieving LMC in this setting.
2. Revealing why increasing model width facilitates LMC. We introduce layerwise exponen- tially weighted connectivity (LEWC), which states that the intermediate- outputs of the merged model can be expressed as an exponentially weighted average of the corresponding outputs of the two original models. Since LEWC implies that the merged model behaves like an ensemble of the two models in terms of predictive performance, it explains why LMC holds. We show that widening the model makes it more likely for LEWC to hold.
3. The role of low-rank structure in achieving LMC. We further show that the low-rank structure of weight matrices plays a crucial role in achieving LEWC. Previous work on permutation-basee model merging has pointed out that low-rankness of weights is essential for LMC (Ito et al., 2025a; Yunis et al., 2024). We demonstrate that this requirement also applies to LEWC by conducting experiments that vary the degree of weight decay. These results suggest that the possibility for LMC to hold depends strongly on properties of the solutions obtained by SGD.
2 BACKGROUND AND PRELIMINARIES
2.1 NOTATION
For natural number k E N. denote [k] := 21, k}. We use bold uppercase letters (e.g.. X) for tensors and matrices, and bold lowercase letters (e.g.. x)) for vectors. Given a tensor X, its vectorized form is written as veo(X), and |X|| indicates its Frobenius (T.2) norm.
2

---

Published as a conference paper at ICLR 2026
Throughout this paper, consider multilayer perceptrons (MLPs) f(x; 0) with L layers, though the analysis can be extended to other neural architectures. The input is æ E R and the parameter set is 0 R where din and dna denote the input and parameter dimensions, respectively. Let & be the representation at the (-th layer, defined recursively as žn = a and ž a (Wiz - be) for l E [L], where is the activation, and Wt, by the weight and bias of layer f. When parameters need to be emphasized, we denote the ε-th layer's output as fæ; 0). The full parameter vector can
be expressed as 0 Diz1 (vec( @ by) where @ denotes concatenation.
2.2 LINEAR MODE CONNECTIVITY (LMC)
Let E Rd be a model and (()) its loss. For two models O, and O,, the loss barrier is defined as: Definition 2.1 (Loss Barrier (Entezari et al., 2022)).
B(O,, 0.) max (C(10, + ^)O,) [AC(O.) +1 (1- AE|0,1]
Intuitively, B(,.. 0%) measures how much the loss increases when linearly interpolating between the two models. If this barrier is nearly zero, we say O, and O,, are linearly mode connected.
2.3 PERMUTATION SYMMETRY AND LMC
Neural networks exhibit permutation symmetry in their parameter space. For the L-th layer of a network with parameters 0, permuting its outputs and compensating at the next layer leaves the input-output behavior unchanged. We denote this transformation by π (0), where x is a pernutation.
Ainsworth et al. (2023) showed that two independently trained wide networks can achieve LMC by permutation symmetries through weight matching (WM), which finds permutations that minimize their 2 distance. Intuitively, if 0. x (0,) for some æ,, then 0, and T(,)) can be seen as almost the same parameters, so their interpolation XO + (1 (),) should preserve performance, where the merging (interpolation) coefficient. This view suggests that models scattered in parameter space may lie in a common loss basin once permutations are accounted for. However, al.,
prior works (Ainsworth et al., 2023; Entezari et al., 2022; Guerrero-Peña et al., 2023; 2025a) indicate that WM requires very large widths (e.g., 32x for ResNet-20, 4x for VGG-16). In contrast, our results show that simply widening the models already improves merged performance monotonically, eventually matching the originals without permutation.
WIDTH EXPANSION FACILITATES HIGH-ACCURRR MODEL MERGING
Ainsworth et al. (2023) experimentally demonstrated that, in permutation-base model merging, sufficiently wide models are required for LMC to hold. In this section, we empirically show that without permutations, widening the models improves the performance of merged models. First, we confirm that widening increases test accuracy and reduces test loss. Next, we demonstrate that applying random permutations does not degrade performance. These results suggest that, once the models are sufficiently wide, permutations are no longer essential for achieving LMC.
3.1 TEST ACCURACY AND LOSS
Figure shows the mean and standard deviation of test accuracies over three independent merges of two models.' ! For comparison, also include results obtained with the permutations discovered by weight matching (WM) Figure 16. Details the models, hyperparameters, and permutation search methods provided in Appendix B. As shown in the figure, widening the models enables merged models to achieve accuracy comparable to the original ones, even without permutations.
Figure 2 presents the test losses of merged models. As observed in Figure 2(a), the test loss does not decrease sufficiently to match that of the original models, although Figure shows that test accuracy
Beecasse LMC holds only when the models are sufficiently widened, our experiments are limited to relatively simple datasets such CIFAR-10 10 and MNIST. In Appendix F.I, we also provide results on a more complex dataset, CIFAR-100 100, confirming that widening the model similarly removes the harrier.
3

---

Published as a conference paper at ICLR 2026
on MNIST
MLPP on FMNIST
VGG-11 on CIFAR-10
ResNet-20 on CIFAR-10
100
Test acc.
Test acc.
Text scc.
Test acc.
75
75
50
50
50
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Interpolation coefficient
Interpolation coefficient 2.
Interpolation coetficient À
Interpolation coefficient
Figure 1: Test accuracies merged models without permutations for different values of the interpolation coefficient X. Even in the absence of permutations, increasing the width multiplier enables the merged models to reach accuracy comparable to the original models (corresponding to and 1).
MLP on MNIST
MLP on FMNIST
VGG-11 on CIFAR-10
ResNet-20 on CIFAR-10
Test less
Test loss
Test Tons
0.0
0.5
0.0
0.5
0.0
0.5
1.0
0.0
1.0
Interpolation coefficient
Interpolation coefficient A
Interpolation coeffcciett À
Interpolation coefficient
(a) Test losses of merged models.
MLP on MNIST
MLP on FMNIST
VGG-11 on CIFAR-I0
ResNer- 20 on CIFAR-10 10
Calibrtted kot bos
Calitattd sot Ioo
Calttrted lext los.
0.5
0.0
1.0
0.0
1.0
Interpolation cocfficient
Interpolation coefficient
Interpolation coefficient 2
Interpolation coefficient
(b) Calibrated test losses of merged models.
Test losses of merged models without permutations. Figure 2(a) shows the original loss values, while Figure 2(b) shows the values obtained by applying temperature scaling.
Figure 2:
Table 1: Barrier values (A = 1/2) with and without the permutation found by WM. Loss barriers use calibrated losses in both settings. Width multipliers are 16x for MLP, 16x for VGG-11, and 32x for ResNet-20. For sufficiently wide models, barriers are small even without the permutation.
Without perm
With perm
Network
Dataset
Acc. barrier (%)
Loss barrier
Acc. barrier (%)
Loss barrier
MNIST
0.519 + 0.225
0.013 + 0.004
-0.027 0.139
-0.003 + 0.003 0.155 + 0.111
MLP
FMNIST
2.467 + 0.685
0.056 + 0.014
4.925 + 4.379
VGG-11
1.308 1.594
0.066 + 0.054
7.000 3.694
0.177 + 0.039
CIFAR-10
ResNet-20
2.694 + 1.493
0.087 + 0.051
5.135 = 2.935
0.173 + 0.099
monotonically increases with width. This discrepancy arises because certain transformations of the output distribution can change the loss without affecting accuracy. concrete example is scaling the logits by an inverse temperature. which alters the cross-entropy loss but leaves the predicted labels unchanged. Since accuracy the primary objective in classification, is reasonable to evaluate the loss under an optimal inverse temperature. Accordingly, we estimated the inverse temperature using 20% the test set and computed calibrated losses on the remaining 80%. The calibrated results are shown in Figure 2(b). With this adjustment, the loss barrier approaches zero width increases, as expected. In this sense, when we state that LMC holds with a zero loss barrier, we also include the case where the softmax is calibrated with an inverse temperature. For reports the barrier values separately for the cases with and without applying the permutation. We find that the barriers remain very small without the permutation.
detailed view, Table 1
4

---

Published as a conference paper at ICLR 2026
3.2 RANDOM PERMUTATIONS
As discussed above, sufficiently wide models achieve performance comparable to the originals even without permutations. This suggests that permutations are not essential once the models are sufficiently wide. To test this further, we applied random permutations before merging two models at 1/2. As shown in Figure 18(a), the merged models still maintain high accuracy, indicating that permutations are not critical when models are sufficiently wide.
4 EXPANDING MODEL WIDTH ACHIEVES LAYERWISE EXPONENTIALLY WEIGHTED CONNECTIVITY
In this section, we aim explain why widening models facilitates LMC. To this end, introduce the concept of layerwise exponentially weighted connectivity (LEWC) in Section 4.1. When LEWC holds, the intermediate outputs of the merged model can be expressed as exponentially weighted combinations of the corresponding outputs from the original models. Consequently, the output of the merged model becomes equivalent to that of an ensemble of the original models, thereby achieving LMC. We then empirically examine in Section 4.2 whether LEWC actually holds at the merging ratio À = 1/2, and show that widening the model makes LEWC more likely to be satisfied. A more fundamental explanation of why LEWC emerges will be provided in the next section.
4.1 LAYERWISE EXPONENTIALLY WEIGITED CONNECTIVITY
To clarify why widening models enables LMC, we introduce the following key concept.
Definition 4.1 (Layerwise Exponentially Weighted Connectivity). Two models with parameters O, and O, are said to be layerwise exponentially weighted connected if, for every layer l E [L] and any [0, 1], we have
fe(æ; 20 + + (1- 1)0,)= = X (e( fe almost surely.
(I)
When Definition 4.1 holds, the intermediate output of the merged model 0, A0, + (1 A)O, layer e is expressed as a weighted sum of the original models' outputs, where the coefficients decay exponentially with depth as x and (1 A) This also applies to the last layer, yielding fu (x; f (z; 0.) + (1- fL):f 0p). For classification tasks, since scaling the logits by positive constant does not change the predicted labels, we divide the right-hand side by A +1- A to normalize the coefficients into weights that sum to one. In this way, the merged model can be interpreted as an ensemble that uses weighted average of the two models' logits with weights / + (1 - )) and (1 - /24 + +(1 - A)4). Thus, in terms of accuracy, LEWC directly implies LMC (i.e., accuracy degradation).
On the other hand, for the loss, when is close to 1/2, the exponential decay factors X and (1 - X) may cause the loss to increase. However, this effect can be mitigated by applying a temperature-sealee softmax with an appropriate inverse temperature. As shown in Figure 2(b), with suitable calibration the loss barrier can also be reduced to nearly zero
Relation to layerwise linear feature connectivity. Zhou et al. (2023) introduced layerwise linear feature connectivity (LLFC), a concept related to LEWC. LLFC explains why LMC holds for permutation-be based methods and spawning. LLFC states that, for each layer, the output of the merged model can be expressed as a weighted average of the outputs of the two original models. As sufficient conditions for LLFC, Zhou et al. (2023) proposed weak additivity for ReLU activations and a commutativity property. our setting, we show in Appendix C that the commutativity property does not hold. This is why, in this work, we introduce LEWC as concept distinct from LLFC.
4.2 EMPIRICAL. VERIFICATION
We next empirically verify whether LEWC holds at À = 1/2. In this case, Equation (1) reduces to fe(æ; (0, + 0x)/2) = (fe))()((x; 0.) + fe (x; 0z)). Accordingly, we measured the cosine similarity between fr (0., + +z)/2) and (fr( 0,) + fz(æ; 0,)) where the factor (1/2)² can be omitted since cosine similarity is invariant to positive scaling. The results are shown in Figure 3.
5

---

Published as a conference paper at ICLR 2026
Cosine similarity
1.0
MLP on MNIST
Cosine similarity
1.0
MLP on FMNIST
Cosine similarity
1.0
VGG-11 on CIFAR-I0
Cosine similarity
ResNet-20 on CIFAR-10 10
0.5
0.5
0.5
0.5
-0.5
-035
103
-0.
-0.
0.0
0.0
0.0
0.0
Layer index
Layer index
Layer index
Layer index
Figure 3: Average cosine similarity between fe( (0. + 02)/2) and (fz 0.) + fe(æ; 0b)) for each layer when test data are fed into the models. For the last layer, cosine similarity computed between the logits. The color of each plot indicates the degree of width expansion. Wider models exhibit higher cosine similarity, making easier for LEWC to hold.
Across all models, increasing the width consistently improves cosine similarity at each layer. In particular, when the width is sufficiently large, the cosine similarity at the last layer becomes close to 1, indicating that the merged model's output is almost identical to that of the ensemble of the two original models. This explains why the merged model achieves high test accuracy.
WHY DOES LEWC EMERGE IN THE WIDE- WIDTH REGIME?
In this section, clarify why widening the model leads the satisfaction of LEWC. First, in Section 5.1, we introduce, as sufficient conditions for LEWC, (1) weak additivity for ReLU activations and (2) reciprocal orthogonality. When (1) and (2) are satisfied, LEWC holds. Therefore, in Sections 5.2 and 5.3, we empirically verify whether conditions (1) and (2) are satisfied, and further reveal that the reasons for their validity are due to the low-rank structure of the weights. In other words, conditions (1) and (2) do not hold when the weights are not low-rank, in which case neither LEWC nor LMC holds. Previous studies (Ito al., 2025a; Galanti et al., 2022) have pointed out that weakening weight decay during SGD training increases the rank of the weights. Therefore, in Section 5.4, we empirically demonstrate that by indirectly increasing the rank of the weight matrices through weakening weight decay, both LEWC and LMC do not hold. This clarifies that the rank of the weight matrices strongly influences the realization of LEWC and LMC.
5.1 SUFFICIENT CONDITIONS FOR LEWC
In this section, introduce, sufficient conditions for LEWC, (1) weak additivity for ReLU activations and (2) reciprocal orthogonality. First, (1) is defined as follows.
Definition 5.1 (Weak Additivity for ReLU Activations (Zhou et al., 2023)). Let 2!(0 and 2,(0) be the f-th layer pre-activations in two models, respectively. These models satisfy weak additivity
for ReLU activations if, for every layer é E [L] and any À E [0, 1]. o(EE(") (1 (1-x)=(") x)=("))
Aa (2) (1 - (z)() almost surely, where a denotes the ReLU activation function.
This property implies that the ReLU activation behaves linearly with respect to the pre-activations along the interpolation path between the two models.
The other condition, reciprocal orthogonality, is defined follows.
Definition 5.2 (Reciprocal Orthogonality). We say that two parameters O,, and O, satisfy reciprocal orthogonality if, for every hidden layer é {2, 3, ......... L}. we have (a)
(b) (a) ker W almost surely. Equivalently, w 2 0 and w(() (a) (b) 2 E ker w and z
(h)
0.
Reciprocal orthogonality means that multiplying the activations input to layer E of one model by the weights layer of the other model yields zero. When both weak additivity for ReLU activations and reciprocal orthogonality are satisfied, we can derive the following theorem:
Theorem 5.3. For two bias-free models 0, and 0,. if Definitions 5.1 and 5.2 hold, then LEWC is satisfied.
The proof of Theorem 5.3 is shown in Appendix E. Regarding the assumption that biases can be ig- nored, in models such as ResNet and VGG that employ batch normalization after every convolutional

---

Published as a conference paper at ICLR 2026
Cosine similanity
1.0
MLP on MNIST
Cosine similarity
1.0
MLP on FMNIST
Cosine similnity
1,0
VGG-I1 on CIFAR-10
Cosine similarity
1.0
RexNet-20 on CIFAR-10
12
XI8
135
0.8
0.8
0.8
0.8
+
2.5
5.0
7.5
5
10
15
Layer index
Layer index
Layer index
Layer index
Figure 4: Average cosine similarity between (( + + and -
where (z
=(") and = are the arcavctivations of the f-th layer of two models. Different colors indicate different width expansion factors. The results indicate high cosine similarity for all layers.
layer, the effect of biases can indeed be considered negligible. Moreover, the exponential decay in activation norms is compensated for by the normalization step. Therefore, from Theorem 5.3, LMC can be achieved when Definition 5. and Definition 5.2 are satisfied. Thus, in the following two subsections, we empirically confirm whether these two conditions actually hold.
5.2 EMPIRICAL VERIFICATION OF WEAK ADDITIVITY FOR RELU ACTIVATIONS
We first verify that ReLU activations behave approximately linearly in sufficiently wide models. Figure 4 shows cosine similarity results used to evaluate ReLU linearity. As seen in the figure, the similarity increases with width, indicating improved linearity.
In the following, we explain from two perspectives why ReLU appears approximately linear when the width increased. The first reason is the effect of the curse of dimensionality due to the increase in the dimensionality of the intermediate layer outputs. Regarding this point, through an analysis using Gaussian random vectors, we clarify the effect of increasing dimensionality on the ReLU function. The second reason is that when the weights of a trained model become low-rank by widening models, the active neurons of the two models will not overlap. From these two reasons, it is desirable that the width is large and the weights are low-rank to achieve the weak additivity for ReLU activations.
Curse of dimensionality on ReLU activations. In high dimensions, two Gaussian random vectors and e yield high cosine similarity between o( + v) and o (u) + av) (approximately 0.93). Theorem 5.4. Let u, U (O, Ia) be two Gaussian random vectors in Rd, Then, with probability at least 35, for every real number satisfyyng e 2 K max(( (v log(2/5), od 1og(2/8))
T (1++) S (a(u) + o(v)) a(
+ 1111
3 + 1 +
(o(u)) + aavillll
((1 - 8) (1 + -e)
where is a constant and K= = 32/33.
As d grows large, /I// Rs 0.93. Thus, in high dimensions, ReLU behaves almost linearly. While real neural network rreavttivatoons are not Gaussian, this suggests that similar effects arise in practice.
0, so the cosine similarity converges in probability to (3/4
Low-rank structure reduces overlap among active neurons. The Gaussian argument implies that a cosine similarity of about 0.93 can be achieved when the dimension is sufficiently large; however, Figure shows cosine similarities higher than 0.93 for wide models, so dimensionality alone does not fully explain the approximate linearity of ReLU. An additional reason is the low-rank structure of weight matrices induced by widening. To illustrate the basic idea, let ={(0) and =(²) be the pre-activations at some layer E in the two models. Let de be the dimensionality at this layer, and for
simplicity assume de is even. Suppose that in =(()) all coordinates except the first de/2 are zero (i.e.. Vi E (ds/2+1, d .....,, dz), 2(a)
0), and in = all coordinates except the last de/2 are zero (i.c., Vi 21, .1. 2. 0). Then we can show that (x)()) =(²)) a (0) + a (z(())
because, for any i, a ((!)) = =((()) i S de/2, and a ( otherwise. On the other hand,
7

---

Published as a conference paper at ICLR 2026
MIPP on MNEE
211
MLEP on FAINST
VGG--1 on CHFAR-ID
linheet-at on CXPA8660
200
Counrr
0255
0.0
0.2
0.00
0.26
0.50
0.7g
0.0
011
0.2
0.1
0.2
6.3
Square 2nd moment
Square of 2nd
Square toot of 2nd
Square 2nd momen
Figure 5: Histogram of the square root of the second (uncentered) moment of ReLU inputs in the
second hidden layer (i.e., V EE We present results for MLPs with width x x16, VGG-11 scaled by x 16, and a ResNet-20 scaled by x32. Most dimensions fall into the leftmost bin, indicating that only a small number of dimensions are active. Results for all layers are provided in Appendix F.7.
MLP en MNSST
MIPP un FMNNST
VGGG-1 on CIFAR-10
RewNes--0 on CIFAR-10
100
Small tipst momeet
Small tepet nornet
Nessocelap
Nassocrlap
0.30
0.50
Sral lepet norrert
Sral lepet morret
Nasonclep
Nasoneslep
0.00
0.00
0.00
0.00
With mathppee
Witth mathgger
Witth mutiger
Witth matiiteer
Figure 6: Proportion of input dimensions with a small square root of the second moment ("Small input moment") and the proportion of non-overlapping large-moment dimensions ("Non-overlap") between two trained models for the second ReLU layer. Here, "Small input moment" refers to dimensions whose second moment is smaller than one hundredth of the maximum second moment across all dimensions, corresponding the leftmost bin in Figure As the model width increases, the fraction of
dimensions grows and the overlap decreases.
similar relation also holds for or (E o Thus, in this case, ReLU function behaves linearly.
From the above considerations, for the pre-activations each layer of the two models, having smaller overlap among the dimensions whose magnitudes deviate significantly from zero makes the linearity of ReLU more likely to hold. In particular, this is more likely when each weight has
low rank. This is because one model's pre-activation is given by 2 -W x and if many
(a)
coordinates of z{( are close to zero regardless of the input, then W(a) must have low rank. the rank of w((a) is large, then when regarding w>(")
linear mapping, the dimension of its output space is large, so the number of coordinates near zero necessarily decreases. Ito et al. (2025a) showed that the model width increases, the rank of each layer's weight matrix becomes relatively small compared to the width, suggesting that widening the model induces low-rank weights and makes the linearity of ReLU more feasible."
We empirically verified whether widening the model reduces, for each layer, the relative number of dimensions with a large second moment of the pre-activations. Figure shows the histogram of the square root of the pre-activation second moment at the second hidden layer, demonstrating that most dimensions have negligible second moment. This indicates that only limited number of dimensions effectively contribute to the model's output. Figure 6 further shows that as the width increases, the overlap between the large-moment dimensions of two independently trained models decreases, supporting the approximate linearity of ReLU.
5.3 EMPIRICAL VERIFICATION OF RECIPROCAL ORTHOGONALITY
We next empirically examine whether reciprocal orthogonality holds. Figure 7 reports the ratio (b) E| B[[W(()) = 1/ // 2[01 Il. As model width increases, this ratio decreases across all layers ex- cept the input, indicating approximate reciprocal orthogonality. To further confirm this, note that, if re-
Loo rankness does not by itself imply coordinate level sparsity, which is the more direct driver of weak additivity. Nonetheless, our experiments indicate that the two are tightly coupled in practice: varying weight decay changes the rank of the weight matrices, and observe corresponding change in coordinate level sparsity of the pre-activations.
8

---

Published as a conference paper at ICLR 2026
on MNIST
MLP on FMNIST
VGG-11 on CIFAR-10
ResNet-20 on CIFAR-I0
1.0
1.0
1.0
1.0
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
12 18
Layer index
Layer index
Layer index
Layer index
Figure 7: Ratio of mean norms E W thh I width suggests approximate reciprocal orthogonality.
(o)
at each layer. The decreasing ratio with
MI.P on MNIST
MI.P on FMNIST
VGG-11 on CIFAR-10
ResNet-20 20 on CIFAR-10
(e d0 wy
0.5
203
24
103
100
122
0.0
0.0
0.0
0.0
2
3
12
Layer index
Layer index
Layer index
Layer index
Figure 8: Average cosine similarity between W
and W z for each layer. The similarity
increases with model width and approaches one, indicating that reciprocal orthogonality holds.
VGG-11 CIFAR-10
ResNet-20 on CIFAR-10
VGG-11 CIFAR-10
ResNet-20 CIFAR-10
100
100
Callestee kon
Callbateed lest loos
36
Teat nec.
Text acc.
50
50
0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
Interpolation coefficient
Interpolation coefficient A
Interpolation coefficient 2.
Interpolation coefficient
Test accuracies of merged models.
Calibrated test losses of merged models.
Figure 9: Performance of merged models for VGG-11 11 and ResNet-20 trained with weak weight
decay (10-1). Note that most our results in this paper used a weight decay coefficient of 0.003.
ciprocal orthogonality and LEWC hold, then applying the weight matrix of one model (e.g., W to the merged model's intermediate activation x should reproduce the corresponding pre-activation of that model (i.e., 2(()). Indeed, w/()) (c)
z (w z + W1
(a) (a)
(a) (b) Xw (a) (a) To
verify this empirically, compute the average cosine similarity between W
and w (a) zg-1
across test data. The results are shown Figure 8. As expected, the cosine similarity increases with model width and approaches one, confirming that the reciprocal orthogonality holds in practice.
From the above, we confirm that reciprocal orthogonality holds in wide models. This is also attributable to the low-rank structure of the weights. For example, if trained model's weight matrices have high rank, then since one model's weight wr(a) at some layer has high rank, [|W(()) z |//ll//| =--|| becomes large regardless of the direction of the input z This implies that reciprocal orthogonality does not hold. Ito et al. (2025a) clarify that the relative rank of weights with respect width becomes smaller for wider models, explaining why reciprocal orthogonality is more likely to hold when the model is wider.
5.4 EFFECT OF WEIGHT DECAY ON LMC
We argued that the improved performance of wide merged models arises because both weak additivity for ReLU activations and reciprocal orthogonality hold, leading to LEWC. As mentioned, these properties are easier to satisfy when weight matrices are low-rank.
Prior works (Timor al., 2023; Galanti et al., 2022; Ito et al., 2025a) empirically observed that widen- ing models and applying stronger weight decay encourage low-rank weight matrices. Conversely, when weight decay is weak, weight matrices tend to have higher rank, making LEWC less likely.

---

Published as a conference paper at ICLR 2026
Cosine sinilarity
1.0
MLP on MNIST
Cosine similarity
1.0
MLP on FMNIST
Cosine similaritt
1.0
VGG-11 on CIFAR-10
Cosine similarity
ResNet-20 on CIFAR-10 10
0.5
0.5
0.5
0.5
-0.5
10.
103
0.0
0.0
0.0
0.0
Layer index
Layer index
Layer index
Layer index
(a) Average cosine similarity between fe( (0. + 0.)/2) and (fe(a;; 0.) + fela;; 02)) 2 for cach layer when
test data are input to models.
Cosine similarity
MLP MNIST
Cosine similarity
1.0
MLP on FMNIST
Cosine similarit
VGG-11 on CIFAR-I0
Cosine similarit
ResNet-20 on CIFAR-10 -00
VVV
0.8
1033
0.8
0.8
+
2.5
5.0
7.5
10
Layer index
Layer index
Layer index
Layer index
(b) Average cosine similarity between o(
+2, )/2) and (a(z)))
+
1/2
MLP on MNIST
MLP on FMNIST
VGG-11 on CIFAR-10
ResNet- 20 on CIFAR-
1.0
B| [
1.0
1.0
1.0
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
Layer index
Layer index
Layer index
Layer index
(c) Ratio of mean norms B| W
(b)
1/ W
for each layer.
Figure 10: Experimental results for trained models with weaker weight decay of 10-4, From top to bottom, the figures show evaluations of LEWC, weak additivity, and reciprocal orthogonality.
Figure shows that merged models trained with weak weight decay indeed exhibit large barriers. To understand this phenomenon in more detail, we examine whether LEWC and its sufficient conditions still hold in this setting. The empirical results are summarized in Figure 10. The impact of weaker weight decay is especially pronounced for VGG-11 1 and ResNet-20. First, Figure 10(a) indicates that LEWC is no longer satisfied in deeper layers when À 1/2. Moreover, Figures 10(b) and 10(c) show that neither weak additivity of ReLU activations nor reciprocal orthogonality holds. Overall, these results show that weakening weight decay breaks LEWC and its two sufficient conditions, highlighting the role of low-rank structure in enabling LMC.
CONCLUSION
We empirically showed that simply widening neural networks improves the performance of merged models, eventually matching that of independently trained models. This behavior differs fundamen- tally from the intuition underlying permutation- exponentially weighted connectivity (LEWC), which arises when weak additivity for ReLU activa- tions and reciprocal orthogonality are satisfied, and we found that these properties hold in sufficiently wide models with low-rank weight matrices. Considering LMC in this distinct setting may provide new insights into both LMC itself and, more broadly, the dynamics of neural network training.
approaches to LMC. We introduced layerwise
Our experiments focused on standard image classification with relatively simple architectures, since LEWC typically requires larger width multipliers than permutation- merging. An important direction for future work is to test whether these phenomena extend to larger-scale models and other modalities.
Our work is analytical rather than a proposal of merging federated learning method. By identifying weak additivity and reciprocal orthogonality as the drivers of permutation-fret LMC, we point to practical directions for developing as training or vermutation-searce procedures that promote these properties, and simple rescaling to offset any C-inducee logit norm decay.
and federated learning methods, such
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This paper presents work whose goal is advance the field of machine learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
REPRODUCIRILITY STATEMENT
The experimental settings for reproducing the results are described in Appendix B. All proofs of the theorems are given in Appendix E.
REFERENCES
Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In Proc. of ICLR, 2023.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred A. Hamprecht, Essentially barriers in neural network energy landscape. Proc. of ICML, pp. 1308-1317, 2018.
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. In Proc. of ICLR, 2022.
Damien Ferbach, Baptiste Goujaud, Gauthier Gidel, and Aymeric Dieuleveut. Proving linear mode connectivity of neural networks via optimal transport. In Proc. of AISTATS, pp. 3853-3861, 2024.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In Proc. of ICLR, 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In Proc. of ICML, pp. 3259-3269, 2020.
C. Daniel Freeman and Joan Bruna. Topology and geometry of half-rectified network optimization. In Proc. of ICLR, 2017.
Tomer Galanti, Zachary S. Siegel, Aparna Gupte, and Tomaso Poggio. Characterizing the implicit bias of regularized sgd in rank minimization, 2022.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P. Vetrov, and Andrew Gordon Wilson. Loss surfaces, mode connectivity, and fast ensembling dnns. In Proc. of NeurlPS, pp. 8803-8812, 2018.
Fidel A. Guerrero-Pere Heitor Rapela Medeiros, Thomas Dubail, Masih Aminbeidokhti, Eric Granger, and Marco Pedersoli. Re-basin via implicit sinkhorn differentiation. In Proc. of CVPR, pp. 20237-20246, 2023.
Markus Haas. A note the absolute moments of the bivariate normal distribution. Economics Bulletin, 38(1 38(1):650-656, 2018
Akira Ito, Masanori Yamada, and Atsutoshi Kumagai. Analysis of linear mode connectivity via permutation-basee weight matching: With insights into other permutation search methods. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025a. URL httpstttss//cc wwwpreveeeeeeeeee////p/e. net forum?id= 1YRkGZZi 9D.
Akira Ito, Masanori Yamada, and Atsutoshi Kumagai. Linear mode connectivity between multiple models modulo permutation symmetries. In Forty-second International Conference Machine Learning, 2025b. URL /tttpps://peneeevieer... net.:/ewwwweeeeeeeeeeeeeeeeeeeee F http://pnnrrriieeeeeoooorrmmmaaa wtw.ooummmidggaaaaai.....1
Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur. REPAIR: renormalizing permuted activations for interpolation repair. In Proc. of ICLR, 2023.
Sungyoon Kim, Aaron Mishkin, and Mert Pilanci. Exploring the loss landscape of regularized neural networks via convex duality. In The Thirteenth International Conference on Learning Representations, 2025. URL htttps://ppenreview... otgpr//reeeeeeeeeeeooooooooooooo http://ooooouoooo0000022272
11

---

Published as a conference paper at ICLR 2026
Jakub Konečný, H. Brendan McMahan, Daniel Ramage, and Peter Richtárik. Federated optimization: Distributed machine learning for on-device intelligence. ArXiv preprint, 2016.
A. Krizhevsky and G. Hinton, Learning multiple layers of features from tiny images. Master's 's thesis, Department of Computer Science, University of Toronto, 2009.
Rohith Kuditipudi, Xiang Wang, Holden Lee, Yi Zhang, Zhiyuan Li, Wei Hu, Rong Ge, and Sanjeev Arora. Explaining landscape connectivity of low-cost solutions for multilayer nets. In Proc. of NeurIPS, pp. 14574-14583, 2019.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, pp. 2278-2324, 1998.
Dawei Li, Tian Ding, and Ruoyu Sun. On the benefit of width for neural networks: Disappearance of basins. SIAM Journal on Optimization, 32(3):722--177 1758, 2022. doi: 10.1137/2MM1394205 URL
https: / /doi. wwwg/1.000........ 10. 137 2M1 394205.
Shiyu Liang, Ruoyu Sun, Jason D Lee, and R. Srikant. Adding one neuron can eliminate all bad local minima. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URL 2018 A11.///0001111111101100050000000
wwwp::/wwwdeiine.e.eeeeeeeeeeeee neurips oa cc/paper_ Files/paper/ wwwppeddeee-aaepeeeeeeoee Paper.pd.
Hong Liu, Zhiyuan Li, David Leo Wright Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training. In The Twelfth International Conference on Learning Representations, 2024. URL https: http:///oppp//eeeeee net/ forum?
idd33xDee@Nooi.
Michael S Matena and Colin Raffel. Merging models with fisher-weighted averaging. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: opwprevieeeeeeeeeeeeeee net net/ wwr.mmrrrmmmmmmm deneeummmiiiiii d= LSK1p aceoC.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera Arcas. Communication-efficee learning of deep networks from decentralized data. Proc. of AISTATS, pp. 1273-1282, 2017.
Vaishnavh Nagarajan and J. Zico Kolter. Uniform convergence may be unable explain generaliza- tion in deep learning. Proc. of NeurIPS, pp. 11611-11622, 2019.
Quynh Nguyen. On connected sublevel sets in deep learning. In Proc. ICML, pp. 4790-4799, 2019.
Quynh Nguyen. A note connectivity of sublevel sets in deep learning, 2021. URL https: a /arxivv. org/ abs abaaabaa21 0210/210..000 08576.
Quynh Nguyen, Mahesh Chandra Mukkamala, and Matthias Hein. On the loss landscape of a class of deep neural networks with bad local valleys. In Proc. of ICLR, 2019.
Alexander Shevehenko and Marco Mondelli. Landscape connectivity and dropout stability of sgd solutions for over-parameterized neural networks. In Proceedings of the 37th International Conference on Machine Learning, ICML'20 20. JMLR.org. 2020.
Berfin Simsek. François Ged, Arthur Jacot, Francesco Spadaro, Clement Hongler, Wulfram Gerstner, and Johanni Brea. Geometry of the loss landscape in overparameterized neural networks: Symme- tries and invariances. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 9722-9732, PMLR, 18-24 Jul 2021. simsek21a html.
URL https: htt.tttss//ppoeeeeeeeeeeeeeeeeee http:::://pccceeeeeppeeee wtt.://ddii........rrreess/s/sso
Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. In Proc. of NeurlPS, 2020.
Sidak Pal Singh, Linara Adilova, Michael Kamp, Asja Fischer, Bernhard Schölkopf, and Thomas Hofmann. Landscaping linear mode connectivity. In Himn-dimensoonal Learning Dynamics 2024:
The Emergence of Structure and Reasoning, 2024.
12

---

Published as a conference paper at ICLR 2026
Nadav Timor, Gal Vardi, and Ohad Shamir. Implicit regularization towards rank minimization in relu networks. In Proc. of ALT, pp. 1429-1459) 2023.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. ArXiv preprint, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Proc. of NeurlPS, pp. 5998 0008-608, 2017.
Luca Venturi, Afonso S. Bandcira, and Joan Bruna. Spurious valleys in one-hidden-layer neural network optimization landscapes. Journal of Machine Learning Research, pp. 1-34, 2019.
Roman Vershynin. Dighs-Dimensionl Probability: An Introduction with Applications in Data Science. Cambridge University Press, 2018.
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris S. Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. Proc. of ICLR, 2020.
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Proc. of ICML, pp.
2022.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. ArXiv preprint, 2017.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. TIES-merging Resolving interference when merging models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: /tttt:::://pppeeeeetee http://pppeeeeeeeeeeeeeeee /
David Yunis, Kumar Kshitij Patel, Samuel Wheeler, Pedro Savarese, Gal Vardi, Karen Liveseu, Michael Maire, and Matthew R. Walter. Approaching deep learning through the spectral dynamics of weights, 2024.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang. Junjie Zhang. Zican Dong, Yifan Du Chen Yang. Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian Yun Nie, and Ji-Rong Wen. A survey of large language models. ArXiv preprint, 2023.
Zhanpeng Zhou, Yongyi Yang, Xiaojiang Yang, Junchi Yan, and Wei Hu. Going beyond linear mode connectivity: The layerwise linear feature connectivity. In Proc. of NeurlPS, 2023.
13

---

Published as a conference paper at ICLR 2026
A ADDITIONAL RELATED WORK
(Linear) Mode Connectivity. A series of works (Garipov et al., 2018; Draxler et al., 2018; Freeman & Bruna, 2017) have shown that independently trained neural networks can often be connected by low-loss nonlinear paths. Nagarajan & Kolter (2019) provided an early observation that, on MNIST, solutions obtained from stochastic gradient descent (SGD) with the same initialization can be connected by linear paths with little increase in loss. Later, Frankle et al. (2020) systematically demonstrated that such linear connections are not universal: whether they appear depends strongly on the dataset and architecture. They also observed that when two models are branched from shared partially trained model, the resulting solutions are almost always connected by a linear path. Moreover, they investigated the link between linear mode connectivity (LMC) and the lottery ticket hypothesis (Frankle & Carbin, 2019).
More recent work has explored symmetry and alignment explanations for LMC. Entezari et al. (2022) conjectured that, once permutation symmetries of hidden units are taken into account, LMC should hold with high probability. Building on this, Ainsworth et al. (2023) introduced weight- matching approach based on bipartite graph alignment, while Guerrero- Penereero-Peaaaee et al. (2023) applied Sinkhorn's 's algorithm as a relaxation to solve the alignment problem more directly. Although number of papers ( (Venturi et al., 2019; Nguyen et al., 2019; Nguyen, 2019; Kuditipudi et al., 2019) have studied nonlinear mode connectivity, theoretical understanding of LMC remains limited. Ferbach et al. (2024) established width width-dependent conditions under which LMC can be guaranteed, assuming independence of weight vectors. Zhou et al. (2023) proposed the notion of layerwise linear feature connectivity (LLFC) and proved that LLFC implies LMC. More recently, Ito et al. (2025a) highlighted the role of dominant singular vectors in parameter space, showing that they are critical to satisfying LMC, especially when alignment via weight matching is applied.
In addition, several theoretical works have analyzed the effect of network width on nonlinear con- nectivity. For instance, Nguyen (2019) proved that in over-paranen parameterizee neural networks with piecewise-linear activations, every sublevel set of the loss is connected and unbounded. Shevchenko & Mondelli (2020) further showed that in wide multilayer perceptrons, SGD solutions can be linked through piecewise-lined paths along which the loss barrier vanishes as width increases. However, to the best of our knowledge, there are no theoretical results directly establishing that larger width makes LMC more likely. This motivates our empirical investigation of that question.
Model Merging. Model merging has been studied in close connection with LMC, as well as in the context of federated learning. The idea of federated learning was introduced by McMahan et al. (2017) and Konečný et al. (2016), where models are trained locally on partitioned datasets. Wang al. (2020) explored permuting local model components prior aggregation, while Singh & Jaggi (2020) proposed an approach to merge models by optimal transport, conceptually related to the weight-matching method of Ainsworth et al. (2023). Although the method of Singh al. (2024) was primarily designed for fusion and empirically underperforms weight matching, it can still be interpreted as an LMC-based technique because it enforces alignment within the same architecture.
While our work focuses on merging models trained from different random seeds, there is a parallel line of research on merging models obtained by fine-tuning from the same fundamental model. A representative example is Model Soups (Wortsman et al., 2022), which showed that averaging weights across fine-tuned models trained under different hyperparameters can improve test accuracy without additional inference cost, contrasting with standard ensembling. Matena & Raffel (2022) extended this idea by weighting models according to the Fisher information matrix, leading to more effective combinations. Yadav et al. (2023) introduced TIES-Merging, which prunes parameters with small updates, resolves conflicting signs, and averages only aligned components. These approaches are primarily designed for merging fine-tuned models of the same pretrained model, not for models independently trained from scratch with different initializations. By contrast, our study addresses this more challenging setting and seeks to shed light on the feasibility of model merging under such conditions.
Relation between Model Width and Loss Landscape. A broad trend in prior work is that increasing width tends to simplify the loss landscape of deep neural networks and makes the set of optimal solutions more connected. Nguyen (2021): Simsek et al. (2021) pointed out that the topology of the optimal solution set can change drastically by adding only small number of neurons.
14

---

Published as a conference paper at ICLR 2026
Specifically, Nguyen (2021) proved that for deep networks, having single wide layer of width n + 1, where n is the number of training samples, is sufficient to ensure that all sublevel sets of the training loss connected. Simsek et al. (2021) further showed that adding one extra neuron per layer is enough to turn discrete minima into a single connected manifold. Liang et al. (2018) showed that adding one special neuron each layer can eliminate all bad local minima, so that every local minimum becomes a global minimum. Subsequent works have also reported plaasetranssiiooniiee behavior as width increases. For example, Li et al. (2022) showed that there exists critical width m" such that when m 2 m", all suboptimal basins disappear and only optimal basins remain. More recently, Kim al. (2025) established that in regularized two-layer networks, once the width exceeds certain threshold, the optimal solution set becomes connected in parameter space. These results collectively suggest that widening changes the structure of the optimal set and can eventually yield mode connectivity, especially under regularization. Building on this line of work, we empirically show that, after calibrating the loss via inverse temperature scaling, the stronger property of linear mode connectivity holds in sufficiently wide networks without permutations. We believe our findings point to new directions for understanding SGD learning dynamics and may also inform future techniques for model merging and federated learning.
B EXPERIMENTAL SETTINGS
This section details the setup used to train neural networks and obtain solutions from stochastic gradient descent (SGD). The experiments were conducted on four benchmark datasets: MNIST (Le- cun et al., 1998), Fashion-MNIST (FMNIST) (Xiao et al., 2017), CIFAR-I0 (Krizhevsky & Hinton, 2009), and CIFAR- 100. For weight matching (WM), we adopted the implementation available in the public repository of Ainsworth et al. (2023))
All training and evaluation procedures were performed on a Linux workstation equipped with two AMD EPYC 7543 32-core processors, eight NVIDIA A30 GPUs, and 512 GB of memory. We used PyTorch 2.,.1,, PoTchh Lightning 2.4.0*, and torchvision 0.20.1 as the software framework.
B.
MODEL TRAINING
MLP MNIST and FMNIST. In line with the setup of Ainsworth et al. (2023), we trained fully connected multilayer perceptron (MLP) with three hidden layers, each containing 512 units. ReLU activations were applied the hidden layers. For both MNIST and FMNIST, training used the Adam optimizer with a fixed learning rate of 1 x 10-3 and a weight decay of 3 x 10-3, The batch size was 512, and models were trained for up 100 epochs. No learning rate scheduling was applied.
VGG-11 and ResNet-20 on CIFAR-10 and CIFAR-100. For these experiments, we employed the source code released by Ito et al. (2025b)². The VGG-11 1 and ResNet-20 architectures followed the implementations described by Ainsworth et al. (2023). During model merging, BatchNorm statistics were re-calibrated using the training data, following the procedure of Jordan et al. (2023). Models were optimized with SGD using a learning rate of 0.4 and a weight decay of 3 x 10-3, -3, The batch size was set to 500, and training was carried out for maximum of 100 epochs. Data augmentation included random 32 x 32 crops and random horizontal flips.
C LAYERWISE LINEAR FEATURE CONNECTIVITY
Zhou et al. (2023) pointed out that the existence of LMC in settings such as permutation and spawning can be attributed to a property called layerwise linear feature connectivity (LLFC). Additionally, Zhou et al. (2023) also showed that two conditions are sufficient for LLFC to hold: weak additivity for ReLU activations (Definition 5.1) and the commutativity property defined as follows:
htttps://pytorddt
"ttrees//ppoorc..ogg///iiio////p index. html hhttps://githubb.cm/e55-
15

---

Published as a conference paper at ICLR 2026
MLP on MNIST
MLP on FMNIST
VGG-11 on CIFAR-10
ResNet-20 on CIFAR-10
1.0
L0
1.0
0.5
K0:123-
26 x
0.5
20.23
11
0.5
x5
0.5
.023
x2
40.35
x2
X35
X1225
82
10
x1
14
160
X25
-05
x4
X84
34
K2 4
28
0.0
0.0
0.0
0.0
3
7
3
Layer index
Layer index
Layer index
Layer index
(a) Results without permutation.
MLP on MNIST
MLP on FMNIST
VGG-11 on CIFAR-10
2e on CIFAR-10
1.0
1.0
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
Layer index
Layer index
Layer index
Layer index
(b) Results with permutations found by WM.
Figure 11: Verification of whether Definition C.1 holds at each layer. For all datasets, we compute (b)
dist( (vet( (/ (ve zi-1 + w( 2(-1), ), veo( (W.(a) 2 + w (h) 2(1) )) at each layer. A smaller value in- dicates a higher degree of commutativity. For reference, we also report the results using permutations found by WM in Figure I (b). Without permutations, the values are very large for all layers except
(a)
(b)
the input layer, implying that the commutativity property almost never holds.
Definition C.1 (Commutativityy, Let W(
(a) and w! (b) be the weights, and = and z; the outputs of the f-th layer. The models On and Og satisfy commatativity if, for every layer f E [7.].
(a) (a) W 2--1 + W (b) (b) (0) (b)
2--1 w! 26-1 + W zt-1 almost surely.
Since this condition can be rewritten (W(") (a) w (b) (z (a)
(b)
x 0. it tends hold when
the weights of the two models are close (i.e., [W (a) w (b) || ~ 0). This precisely the objective of WM methods, which explains why WM encourages the emergence of LLFC.
We confirmed in Section 4 that LEWC holds when > 1/2. However, we cannot rule out the possibility that LLFC also holds. It is possible that the cosine similarity in Figure 3 approaching is due to the validity of LLFC. To exclude the possibility, we confirm that the commutativity property, which is one of the sufficient conditions for LLFC, does not hold in our settings. Figure I1(a) presents experimental results on whether the commutativity property holds across layers for all models. For comparison, we also show the results when applying the permutations discovered by WM in Figure (b..). Following prior work, we evaluate the difference between the left- and right-hand sides For ResNet-20, we compute this only for the first convolutional layer of each block. As the results show, without permutations, the commutativity property hardly holds, indicating that LLFC is unlikely to be satisfied in our settings.
of Equation (2) using dist(1r, y) = || - w/l² /||/==/=1l|/)
C.I DIFFERENCE BETWEEN LLFC AND LEWC
We empirically confirmed that, in LMC without permutations, LLFC does not hold whereas LEWC does. In this subsection, we provide a qualitative explanation for why LLFC fails even though LEWC holds.
key difference between LLFC and LEWC is that LLFC a property that arises when the weights of two models are sufficiently close, whereas LEWC is a property that arises when the weights of two models are different, namely approximately orthogonal. This can be understood by focusing on the difference between commutativity and reciprocal orthogonality, which are the respective sufficient conditions that distinguish LLFC from LEWC.
Below we explain this point more concretely using equations. Let be O, and 0, the two models, and for simplicity assume that biases can be ignored. Commutativity for > 1 requires
(b) w(()) (a) 2 w! 2- w(() 2
W! (b) 2 (W!") w )={" --( 0.
=b)
16

---

Published as a conference paper at ICLR 2026
MLP MNIST
MLP FMNIST
VGG-11 CIFAR-10
ResNet-20 CIFAR-10
100
100
100
100
Accurary
Accurayy
Accuracy
Accuracy
50
XO.121
50
50
X0.125
50
v0.05
v2
20.35
22
X36
X0.05
X2
XI0
X5 wis
.0.
-0.5
205
24
2234667
9011
2345677 10L1
2345677
1011
2
1011
Lineer coeflicient.
Linear coefficient.
Lineur voeflicient A
Liurar cuellitient X
(a) Test accuracy of the perturbed model Og + Adu/ ||u|l
obtained by moving O, along a random direction.
MLP MNIST
MLP FMNIST
VGG-11 CIFAR-10
ResNet-20 CIFAR-10 10
Test loes
0.50
Test loss
Thst frees
Test loss
1.5
1.0
X05
1.0
0:25
0.5
90111
23 1545677 1011
1566666 9011
Linear coeffccint
Linear coefficient A
Linear coefficient A
Linear coefficient 2
(b) Uncalibrated test loss of the perturbed model 0. Adu/ obtained by moving 0., along random direction.
Figure 12: Test accuracy and test loss of models perturbed along a random direction.
This condition clearly holds when all weights match, that is, when W!! al. (2023) stated that this explains why WM find permutations that make LMC hold, since the objective of WM is to search for permutations that minimize the distance between the weights of two models. In spawning, is also known that LMC holds between solutions obtained by branching from partially trained initialization and then continuing SGD independently. This is because the first training steps before branching keep the two models close, which promotes commutativity. Zhou et al. (2023) experimentally verified that commutativity holds in the spawning setting as well. In contrast, reciprocal orthogonality clearly does not hold when the two weight matrices match.
W 0. Zhou
0, but if the weight matrices are equal and biases are negligible, then the activations are also equal, and these conditions are not satisfied. This shows that commutativity requires the two models to be sufficiently close, whereas reciprocal orthogonality requires them to be sufficiently different, namely approximately orthogonal, so the two requirements are incompatible. For example, in Figures 7 and 1 (a) we showed that without permutations, commutativity fails while reciprocal orthogonality holds, which reflects that the two models remain sufficiently distinct. Conversely, when models aligned using WM discovered permutations, or in the spawning setting, the weight matrices are expected to be close. In such cases, commutativity is likely to hold, whereas reciprocal orthogonality is unlikely to be satisfied.
Specifically, it requires w/()) (b) 0 and W
(b) 2((
This incompatibility implies that the reason for LMC identified in our paper is fundamentally different from the LLFC based explanation of LMC in Zhou al. (2023). Intuitively, under commutativity, permutations make the two models sufficiently close, so the merged model also remains close to each of the original models, and its output stays similar theirs. In contrast, in our explanation, the two models have weight matrices that are highly different, namely approximately orthogonal, which allows the merged model to embed the two functions independently.
This distinction also explains why inverse temperature calibration is necessary under LEWC. LLFC requires the two weight matrices to be close, so the norm of the merged weight matrix is similar to that of the original models. In contrast, LEWC requires the two weight matrices to be orthogonal, so the norm of the merged weight matrix is strictly smaller than that of each original model. Since this reduction occurs at every layer, the output norm decays exponentially with depth. This provides the reason for the exponential decay of activation norms under LEWC.
D RELATION TO LOSS LANDSCAPE FLATNESS
We showed that LMC becomes easier to satisfy as the width increases. One possible alternative explanation is that widening simply improves the flatness of the loss or accuracy landscape, and that
17

---

Published as a conference paper at ICLR 2026
Cosine similarity
1.0
MLP MNIST
Cosine similarity
1.0
MLP FMNIST
Cosine similority
1.0
VGG-11 CIFAR-10
Coeine simiibriit
1.0
ResNet-20 CIFAR-10
0.5
-3135
x1
x6
0.5
N015S
vi
KS
0.5
50
155
01.5
-2335
x2
x16
50.5
x36
20.55
200
x5
X32
0.0
0.0
0.0
0.0
0.5
1.0
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Linsur coeflicient
Linear coeflicient.
Liurar cuellicient.
Lineaa euelTiiient
Figure 13: Cosine similarity of hidden layer activations between Og and the perturbed model O2 + du du/l|ull.
LMC emerges as consequence of this increased flatness. In this section, we examine this possibility and show that a mere improvement in flatness is insufficient to explain our observations.
D.1 EMPIRICAL EVALUATION OF FLATNESS
As simple proxy for flatness with respect to width, we evaluate how test accuracy and test loss change when we perturb trained model along a random direction. Let 0u and A, be two trained models. We first compute their 2 distance d = ||0, Bell Although d could in principle grow with width, in our setting weight decay prevents it from increasing substantially. We then generate Gaussian random vector and normalize it to unit length, and use a perturbation of the same norm as the inter model distance, namely du/||2/ This construction is intended to compare random perturbations at the same scale as the distance between two independently trained models. For each [0, 1], we measure the accuracy and loss of the perturbed model 0, + Adu/ ///w/l We expect performance to degrade as À approaches 1, and we interpret the rate of degradation flatness. The results are shown in Figure 12. As the width increases, both test accuracy and test loss become almost insensitive to the perturbation. In particular, even À 1, the degradation becomes negligible after a modest increase in width. This indicates that this flatness proxy improves rapidly with width.
proxy for
D.2 WIYY DOES THE FLATNESS IMPROVE WITH WIDTHI
Intuitively, one might expect that moving model parameters random direction would quickly harm accuracy and loss. However, Figure 12 shows that widening significantly suppresses this degradation, even more so than in the interpolation between two independently trained models shown in Figures and 2. We attribute this to the nature of the random perturbation. When the width is large, a perturbation with fixed overall norm spread over a much larger number of parameters, so the per parameter magnitude becomes relatively small. As result, the perturbed model behaves similarly to the original one.
More formally, let We denote the weight matrix of the original model at layer E, and let W denote the random perturbation added to that layer. The perturbed weights are W W Ignoring biases, the pre-activation becomes (We + W = W
W/ Thus, the change in output induced by the perturbation is represented by W z The i th component of this term is Σ, W W Since W/ is independent of z and its total norm is fixed to d, the typical magnitude of each entry Wi.,, decreases as the width increases, because the same norm distributed across more parameters. Moreover, due to the effect of weight decay, the inter model distance d does necessarily grow with width. Therefore, E; W 1t11 is expected to approach zero as the width increases.
To validate this intuition, we measure the cosine similarity of hidden layer activations between the original model 0, and the perturbed model 0. + du/l|uu].. The results are shown in Figure 13. The similarity increases with width, supporting the view that the apparent improvement flatness under random perturbations is largely explained by the diminishing effect of Gaussian noise in wide networks.
D.3 FLATNESS ALONE CANNOT EXPLAIN OUR FINDINGS
As a comparison to the random perturbation setting, we examine the cosine similarity of hidden layer activations between two independently trained models O,, and O,, shown in Figure 14, Even though
18

---

Published as a conference paper at ICLR 2026
Costne similority
1.0
MLP MNIST
Cosine similaaity
MLP FMNIST
Cosiie similority
1.0
VGG-11 CIFAR-10
Coséne: simniburity
1.0
ResNet-20 CIFAR-10
0.5
0.5
0.5
01.
0.0
0.0
0.0 0.0
0.0
0.5
1.0
0.0
0.5
Linear coefliciens
Limesr coeflicient.
Linrar cuellicient
Linear cuefftient
Figure 14: Cosine similarity of hidden layer activations between O, and Os.
the norm of the random perturbation in Figure 13 equals the model distance between O₂ and 0b. the activation similarities exhibit qualitatively different behavior. This suggests that probing flatness via random perturbations does not capture the relationship between independently trained models produced by SGD.
Finally, note that if LMC were explained solely by increased flatness, then the exponential decay of activation norms predicted by LEWC would also need to follow from flatness, which does not. Moreover, if flatness were the only reason, the merged model should achieve test loss comparable the original models even without inverse temperature calibration. However, Figure 2 contradicts this. These observations indicate that improved flatness alone is insufficient account for the emergence of LMC with increasing width.
E PROOF
E.1 MAIN THEOREM
Theorem E.1. For two bias-free models On and 0b, if Definition 5.1 and Definition 5.2 hold, then LEWC is satisfied.
Proof. We prove the claim by induction on the depth f. For the base case 1, the statement holds trivially. Now assume l > 2 and that Equation (1) holds for all layers prior to f. The left-hand side of Equation (1) can be written as
fe(æ; A0 + (1 - A)O,) = o (AW +1 w) fe- +(1 ()0,)
(a(W + (x + (z= 6b))
-No(W (x: 0.)) (1- fe=
-ff((z;; +1 - fexx;
Thus, the statement holds for layer l, completing the induction.
E.2 PROOF OF THFOREM 5.4
We now provide the proof of the following theorem. Theorem E.2. Let u. U
\O Ia) be two independent Gaussian random vectors in Rdd Then, with probability at least 1 36, for every real number satisfying e 2
K mmx ( log(2/6), ai log(2/6)). we have
(o(u) + a(v)) a(u To(u +v)
+ 1 +
(1++) (1+
(o( + [a w)]l
V e) (1+
where e is an absolute constant and K = -2/3.
To this end, we prepare several auxiliary results. First, we recall the definitions sub-Guuseee Gaussian and sub-exponential random variables.
19

---

Published as a conference paper at ICLR 2026
Definition E.3 Bub-Gaussian random variable (Vershynin, 2018)), A random variable X is called sub-Gaussian if there exists a constant C such that, for all 2 0,
Pr([-] 2 I 2exp(-t2/CC
The sub-Gaussian norm is defined as
|X := inf {c 0: E exp x² x²/c²) /c²) 2}
Definition E.4 (Sub-exponential random variable (Vershynin. 2018)), A random variable X is called sub-exponential if there exists constant C > 0 such that, for all f 2 0,
Pr[[ 2 t) - 2exp(-t/C).
The sub-exponential norm is defined
| = inf (t > 0 E exp (1/t) 23.
standard that if is sub-Gaussian, then ||X 11,, || Moreover, for two sub-Gaussian random variables X and Y. we have ||XY| ll I || 11 ||llez
We also need the following classical identity for correlated Gaussians.
Theorem E.5. Let (x. y) be jointly Gaussian random variables with zero mean, unit variances, and correlation p. Then
Fla(x)o(y)) = 1 (0- 2 (VI p² paresin p)
E
Proof.
Note that (x) = (x + [x1)/2. Expanding gives
(z (v)) [x =lyl | + |x||y/]-
We have BB[zy] = p. For the cross terms, using B| y] py. we obtain
B[ E [| w]] /|E|y|yel |/ O,
since + vlyi an odd function under the symmetric Gaussian distribution. Similarly B[<x]]] = 0. Hence,
Blo(r)a(w) = ( Bb>lll)
It is classical fact on absolute moments of Gaussians (Haas, 2018) that
E E|rlyyl
π (VI 111-p= p² + arcsin
which completes the proof.
Proof of Theorem 5.4. Writing the cosine similarity elementwise, we have
(a(u) (v)) ou +) ( + (v) (u
v/)) VVE, +,) (o(u) ((
E(o(u))
By Theorem E.5, straigthfooowward calculations yield
(((((u)) (w;))) u,)]
_ + 1 Elo(u + = =1, B((a(u)+ + 1
Moreover, we can bound the sub-exponential norms:
[I(a(u) + (r + v.)l |lo(u,) + o(v))ll, Ilo(u + will 2 (l(o()))lv l|u. valles < $/2
Similarly, |ιo + l < 32/3 and (I((w))) + (v:))) < 32/3. Hence each of these random variables sub-exponential. Applying Bernstein's 's inequality (Vershynin, 2018) with K 32/3, we obtain
Pr 1/2
(o()) (o( +o(u))a(u u.) - - 21 2exp (-odd-odmii min e k2 )))
(3)
20

---

Published as a conference paper at ICLR 2026
VGG-11 on CIFAR-10 100
ResNet-20 CIFAR-
VGG-11 on CIFAR-100
ResNet-20 on CIFAR- 100
Calle loo
Calle ans Foo lno
Test nec.
50
Test Test
50
5.0
XIE
22
18.2
X26
2.3
2.5
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Interpolation cnefficient A
Interpolation coefffciint
Interpolation coefficient A
Interpolation coefficient
(a) Test accuracies of merged models.
(b) Calibrated test losses of merged models.
Figure 15: Performance of merged models for VGG- and ResNet-20 trained on the CIFAR-100 dataset.
Pr
Σ (o()) -od
(1 1/7) 21) s 2exp (eexp-cd min
R x)
(4)
Pr 1 E ol; + 0;)² -1/2-/1 2t s 22cxp( 2exxp(cddmin k kx) (1:n)
(5)
where e a constant.
Now set d 2 2 min ////KKK/K)K) 1/K)). Equivalently,
(V 2 log(2/8), da log(2/6)
1 2 K max
Letting denote this bound, equation 3 implies that with probability at least - 6,
+ -+--ca - à, "(o(us) + (v:))) + 0;) s 3 +
Analogous statements hold for equation 4 and equation 5. Combining these inequalities yields
1
[( vz)
3 + 1 +
3
((1 c)(1 1/m+C] VOE, olu + +x)))) (1:)))
v(1 () 1/
with probability at least 36, completing the proof.
F ADDITIONAL EXPERIMENTAL RESULTS
F.I CIFAR-100 DATASET
In this work, we primarily focused on experiments with relatively simple datasets such as MNIST and CIFAR-10. This choice is due the fact that LMC does not hold unless the models made sufficiently wide, which makes verification difficult on more complex datasets. In this section, we present experiments conducted CIFAR-100, more challenging dataset, to investigate whether LMC can also be achieved in this setting. Figure 15 reports the accuracy and calibrated test loss when changing the merging ratio A. We observe that increasing the model width leads monotonic decrease of the barrier in both metrics, eventually eliminating it. Notably, while previous permutation- based merging methods required that the original models achieve sufficiently high test accuracy and low test loss in order for LMC to hold, our permutation free approach demonstrates that LMC can be achieved even when the models' performance is comparatively modest.
F.2 PERMUTATION- BASED METHOD
The test accuracy values obtained by merging with the permutations found by WM shown in Figure 16. While the use of permutations improves performance compared to the case without permutations (Figure 1), the gap nearly vanishes as the model width becomes sufficiently large.
21

---

Published as a conference paper at ICLR 2026
on MNIST
MLP on FMNIST
VGG-11 on CIFAR-10 10
ResNet-20 on CIFAR-10 10
100
Tost acc.
Text acc.
Test acc.
50
50
Text
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Interpolation coefficient
Interpolation coefficient 2.
Interpolation cocfficient
Interpolation coefficient
Figure 16: Test accuracies of merged models with permutations found by WM.
Acouacy hariier
100
MI.P on MNIST
Accuracy bonier
100
MI IP on FMNIST
Accuracy barikr
VGG-11 11 on CIFAR-10
Aucuracy harker
Res 20 on CIFAR-10
27
Width multiplier
Width mulipplier
Width mulliplier
multtipieer
(a) Accuracy barrier.
MLP on MNIST
MLP on FMNIST
VGG-11 11 on CIFAR-10
ResNet- 20 on CIFAR 10
barreer
hameer
Las horree
prrm
multipler
multiplaer
multiplier
multiplerr
(b) Calibrated loss barrier.
Figure 17: Accuracy and calibrated loss barriers with and without the permutations identified by WM.
MLP on MNIST
MLP on FMNIST
VGG-11 on CIFAR-10
ResNet-20 on CIFAR-10
secuney
100
Test aoanacy
100
Telt seconay
2
Width multiplier
Width multiplier
Widti multippier
Width mustippier
(a) Test accuracies of merged models with random permutations.
Barter (tfeeens
MLP on MNIST
Barter difference
MLP on FMNIST
Burier diffeence
VGG-11 on CIFAR-10
ResNet-20 on CIFAR-10
Width multiplier
Width multiplier
Width multiplier
Width multiplier
(b) Difference calibrated loss barriers between random permutations and the permutation identified by WM.
Figure 18: Effects of random permutations accuracy and loss barriers.
F,3 LOSS AND ACCURACY BARRIERS
Figure 17 shows how the barrier changes with increasing width with and without the permutation found by WM. We observe that applying the permutation tends to reduce the barrier regardless of the width. However, when the width is sufficiently large, the barrier becomes nearly zero regardless of whether the permutation is used.
F.4 RANDOM PERMUTATIONS
For two models O,, and Op trained by SGD, applied randomly chosen permutations n. and to each model, and merged the resulting models n. (0.) and nT(6)). The corresponding test accuracies shown in Figure 18(a). We observe that, when the width is sufficiently large, high test accuracy is obtained regardless of the choice permutations. Furthermore, Figure 18(b) shows the difference in calibrated loss barriers between using random permutations and using the permutation identified
22

---

Published as a conference paper at ICLR 2026
[mvarse temperatie
MI. P MNIST
Invease semperiitee
MI.P FMNIST
Invaree temperature
VGG-11 CIFAR-10
Invesse semperinuee
ResNet-20 CIFAR-10
1.0
1.0
1.0
1.0
0.5
0.5
0.5
0.5
0.0
0.0
Width multiplier
Width multiplier
Width multiplier
Width multiplier
Figure 19: Inverse temperature used for logit calibration when À = 1/2.
El [.EE) x. Rull
on MNIST
El Ielle Fall
MLP on FMNIST
VGG-11 on CIFAR-I0
El EE sull
ResNet-20 20 on CIFAR-10
0.5
0.5
0.5
0.5
0.0
0.0
0.0
4 10121441618 0.0
ReLU index
ReLU index
ReLU Index
ReLU index
Figure 20:
The ratio activation norms between the merged model and the original models,
fe(;; (0, (t)//)]l/B(jf(((:: 0.) + fe(x;; 0b)],, for A 1/2. The test data are fed into the models.
by WM. When the width is large enough, the barrier values remain nearly unchanged regardless of which permutation applied.
F,5 INVERSE TEMPERATURE
We present the inverse temperature values used for calibration when À == 1/2 in Figure 19. This inverse temperature corresponds to the factor by which the logits are divided during calibration. As the results indicate, under sufficiently wide conditions, the inverse temperature takes small values, compensating for the reduction the logit norms caused by LEWC.
F.6 EXPONENTIAL DECAY OF ACTIVATION NORMS
In Definition 1, we showed that the norm of the merged model's output decreases exponentially. In this subsection, we experimentally verify whether this phenomenon actually occurs. The results are presented Figure 20. For the MLP, as the depth increases, the activation norms of the merged model decrease exponentially, which is consistent with the prediction from LEWC. In contrast, for VGG-11 and ResNet-20, the norms do not decrease exponentially. This is presumably because batch normalization compensates for the variance.
F.7 SECOND MOMENTS OF RELU INPUTS FOR ALL LAYERS
In this subsection, we present histograms of the square root of the second (uncentered) moment of the ReLU inputs for all layers of each model. The results correspond to MLP with width scaled by 16x, VGG-11 scaled by 16x, and a ResNet-20 scaled by 32x. The histograms are shown in logarithmic scale. In addition, the values shown to "Small input std." indicate the proportion of input dimensions that fall into the leftmost bin. From Figure 21, can be observed that in every layer, the vast majority of dimensions have very small second moment around zero.
Figure 21. The vertical axis is plotted
F.8 OVERLAP RATIO OF ACTIVE NEURONS WHEN USING PERMUTATIONS
Figure 22 shows the overlap ratio of active neurons the second ReLU layer when applying the permutations identified by WM. This figure corresponds to Figure 6, where permutations are not used. When permutations are used, the non overlap ratio of active neurons clearly decreases, indicating that most neurons with large output second moments overlap between the two models. We attribute this to the fact that WM searches for permutations that reduce the L2 distance between the weight matrices of the two models, which in turn tends to align their activation patterns.
23

---

Published as a conference paper at ICLR 2026
F.9 DEPENDENCE ON WEIGHT DECAY
Figure 23 shows the barrier values for permutation-free merging while varying the weight decay coefficient from 0 to 0.003. For sufficiently wide models, smaller weight decay results smaller barriers.
F.10 SECOND ORDER OPTIMIZATION
the main experiments, we used first order optimizers such as SGD and Adam. Here, we additionally examine the case where second order optimization method is employed. Specifically, we adopt Sophia as a second order optimizer (Liu et al., 2024). The barrier results are shown in Figure 24. For reference, also report the test accuracy and calibrated loss of the trained models in Figure 25. The training conditions, including hyperparameters, are kept the same as the first order optimization setting. Therefore, the weight decay is fixed to 0.003.
From Figure 25, we observe that using a second order optimizer improves both accuracy and loss. However, the barriers are smaller when using first order optimizers. With second order optimization, the barriers remain clearly larger than zero for all models except for the MLP trained on MNIST. Nevertheless. the barrier decreases as the width increases, suggesting that LMC may hold when the width is sufficiently large, Since we did not retune hyperparameters for Sophia, it is also possible that using larger weight decay or a different learning rate would make LMC more likely to emerge under second order optimization.
F.11 EFFECT OF SKIP CONNECTIONS AND BATCH NORMALIZZATION
To clarify the effects skip connections and batch normalization, we prepared two variants of the MLP: with skip connections and with batch normalization layers. For these models, examined the barrier values and how easily reciprocal orthogonality is satisfied. Skip connections were added by summing each intermediate layer input with its output, for all hidden layers except the input and final layers. The resulting barrier values are shown in Figure 26, where the loss barrier is computed using calibrated loss. As shown the figure, adding skip connections or batch normalization leads to smaller barriers.
To assess reciprocal
also evaluated the mean norm ratio (a) = |/E|| (/ {()) for each layer, as reported in Figure 27. The results indicate
orthogonality,
E| |
that the presence absence of skip connections batch normalization does not substantially change this ratio.
USE OF LARGE LANGUAGE MODELS (LLMS)
In preparing this manuscript, we made use of a large language model (ChatGPT-5, developed by OpenAl). The model was employed exclusively check the fluency and naturalness of English sentences drafted by the authors. All technical content, mathematical derivations, experimental design, and scientific claims were created and verified by the authors. The authors carefully reviewed and edited all model outputs to ensure accuracy and clarity.
24

---

Published as a conference paper at ICLR 2026
Layer
Layer 2
Layer 3
10"
Small Inpot moment,
80.3%
Connt
10
Small inpat moneen
96.
10
Small isput moment
59.6%
10
10
101
0
0.0
0.1
0.2
Square rool 2nd mommeul
Square rook of 2nd woment.
Square root 2nd moment
(a) MLP on MNIST
Layer
Layer
Layer 3
Count
102
Small input momennt
74.3%
Count
10a
Small imput
84.8%
10
Small imput mecnent 69.5%
10'
10+
10
20
30
0.0
0.2
0.1
Square root of 2nd tooment
Square root of 2nd monenus
Square root 2nd moment
(b) MLP on FMNIST
Layor
Layer
Laver 3
Liger
107-
Staill 95.8%
10-
Small 90.0%
Small imput 95.15
10F
Small 59.8%
Court
100
Chead
0.01
012
0.00
0.10
0.48
0,000
0,025
0.00
0105
0.10
Sygrene must of Dack moneent
Seguan mosk of 2nd momeee
Segare noct of 2ud mumoer
Segtare rine of 2nd mnvtree
Layer
Layer
Laper
Layer
lnput
Swail inpul 15.
laput
Small lapul moumers
91.0%
Count
to-
96.15
Curr 10
105
10P
10F
D1N0
0.145
0.000
0,076
0.10
0.01
0.22
0,000
DIMG
0,300
Square ro0t of 2nd moncett
Square noot of 2nd moment
Square roct of Snd nomert
Sqgucee noot 2nd momernt
(c) VGG-11 on CIFAR-10
Layer
Layer
Layer 3
Layer
Count
20
Small input momentt
Count
Small iuput momnent
Cocnt
10a
Small input momertt
Connt
10
Small input momsen
58.2%
98.8%
83
98.4%
202
10a
0.0
0.1
0.2
0.00
0.05
0.10
0.00
0.05
0.10
0.00
0.05
0:10
Square of 2nd moment
Square 2nd moneel
Square root monent
Squace of 2nd moment
Layer 5
Layer
Layer 7
Layer 8
Count
10
Smoll inpul momeetl
81.8%
Count
10a
Small impat: moueent
99.2%
Count
10
Small imput ancoeed
60.3%
Comnt
10
Small input moneent
86.0%
702
10
0.00
0.06
0.10
0.00
D.05
0.00
0.05
0.00
0.01
0.02
Square root 2nd moment
Square root moment
Square roor, moment
Squar root of 2nd morent
Layer 9
Layer 10
Layer 11
Layer 12
Count
20a
Semill Input momentt
82.6%
Count
10a=
Small iopat mornent
95.5%
Count
10
Small impus
700655
Count
10
Small input morsed
90.8%
20
10a
109
0.000
0:05
0.10
0.05
0.10
0.0
0.00
0002
Square rool of 2nd mocenn
Square root of 2nd moment
Square root of 2nd moment
Squace rool of 2nd momennt
Layer 13
Layer 14
Layer 15
Layer 16
Coust
10
103
Smmil impul momenll
= 77.5%
Count
10p
10
Small ispat mounent
92.8%
Count
10a
10
Small impui momentt
18.1%
Connt
10
Small inpul momeent
98.1 98.1%
0.0
0.1
6.000
0.0225
0.050
0.0
0.1
B.0
0.2
Square root of 2nd moment
Square root of 2nd moment
Square roor of 2nd moment
Square root of 2nd morment
Layer 17
Layer
Layer 19
10a
Scmll input moment
10a
Small impat mornent
10
Small Input moment
Count
76.5%
Count
87.4%
Count
62.4%
10a
10a-
10P
0.0
0.5
I.D
0.00
0.05
0.10
0
Square of 2nd momennt
Square
rool 2nd momoet
Square rost. 2nd
(d) ResNet-20 CIFAR-10
Figure 21: Histogram of the square root of the second (uncentered) moment for each dimension of the ReLU inputs across all layers.
25

---

Published as a conference paper at ICLR 2026
MLP en MINST
MIPP un FANNST
VOG-11 ou CIFAR-10
RewNet-21 on CIEAR-10
110.
Suall lugat unALT
Suall lupal UNALAR
Novsoeelap
Phopetaa
Newsoerlap
Phopetza
Scral lepur mocrett
Notooedapp
Scual lpdd usseasa
Nanoeclep
0.00
0.00
0.00
0.00
Willh mallppar
Willh anllplrr
Wallh ankipar
Wall ankipler
Proportion of input dimensions with small square root of the second moment ("Small input moment") and the proportion of non-overlapping large moment dimensions ("Non overlap") between two trained models for the second ReLU layer, when using permutations found by WM.
Figure 22:
Accuracy bartier
100
MLP on MNIST
100
MLP on FMNIST
Accuncy barrier
100
VGG-11 on CIFAR-10
Loss burrier Avcuracy barier
100
ResNet-20 on CIFAR- 10
50-
Accuracy
-0
50
50
10
10
10
10
weight_decrg decay
weight_decay
weight_decay
weight_decay
(a) Accuracy barrier.
MLP on MNIST
MLP on FMNIST
VGG-11 on CIFAR-10
ResNet-20 on CIFAR-10 10
Loss burter
Loss burrier
barrier
135
0.0
10 10
10
10 10 +
10
0
10
10
10
10
weight_decay
weight_decan
weight_decay
weight_decaay
(b) Calibrated loss barrier.
Figure 23: Accuracy and loss barriers when varying the strength of weight decay.
Accuracy barrier
100
MILP on MNIST
Accuracy barrier
100
MI.P on FMNIST
Accurncy barrier
VGG-11 on CIFAR-10 100
Accuracy barrier
ResNet-20 on CIFAR-10 100
mndedee Jeat wriis
.50
50
50
50
f6-
10p
10
10
10
10
10"
10
10
Width multiplier
Width multiplier
Width multiplier
Width multiplier
(a) Accuracy barrier.
MLP on MNIST
MLP on FMNIST
VGG-11 on CIFAR-10
ResNet-20 on CIFAR-10
Loss barrier
Lows barrier
Loss barrier
Loss barree
10
10
10
10
10"
10
10
10
10
10
10
Width multiplier
Width multiplier
Width multiplier
Width multiplier
(b) Calibrated loss barrier.
Figure 24: Accuracy and loss barriers with and without second-order optimization (A 1/2).
26

---

Published as a conference paper at ICLR 2026
MLP MNIST
MLP FMNIST
VGG-11 CIFAR-I0
ResNett-2 on CIFAR-
100
100
100
Accuracy
Accuracy
Accuracy
Accuracy
Farcontor
16
10"
10
10"
10
10P
10
10"
10 Width multiplier
Width multiplier
Width multiplier
Width multiplirr
(a) Accuracy.
MLP on MNIST
MI.P on FMNIST
VGG-11 on CIFAR-I0
ResNet-20 on CIFAR-10 10
Loss
0.2
Loxs
0.5
Loss
0.50
Loss
0.3
0.4
0.2
0.3
0.25
10
10
Width multiplier
Width multiplier
Width multiplier
Width multiplier
(b) Calibrated loss.
Figure 25: Accuracy and loss of trained models with and without second order optimization.
MNIST
FMNIST
MNIST
FMNIST
Accuracy battlet
100
Acc urrecs barriet
Loss bertier
MTP
AFF
50
Loss
10
10
10
10
10"
10
10P
Width noltiplier
Width multippier
Width multiplier
Width multiplier
(a) Accuracy barrier.
(b) Calibrated loss barrier.
Figure 26: Accuracy and calibrated loss barriers of models trained with and without skip connections or batch normalization.
MNIST
FMNIST
MNIST
FMNIST
1.0
1.0
1.0
BIW, a.ll (B WWW.
1.0
R(W,A WWA.AIPR W.
W. F. (//
W. (172) W.
v0.32
0.5
20,3
x4
0.5
0.5
28.3
0.5
0.0
0.0
0.0
0.0
3
3
Layer index
Layer indes
Layer index
Layer index
(a) MLP with skip connections.
(b) MLP with batch normalization layers.
W| (0) (b) W/ (a) (a)
Figure 27: Ratio of mean norms E| connections or batch normalization.
for each layer, for MLPs with skip
27


Reviews:
[{"review_id": "6TWoT72YA6", "reviewer": "Reviewer_8AS9", "summary": "In the context of model merging and parameter permutations, this work shows that contrary to prevailing thought, increasing network width  is sufficient in and of itself (without permutation alignment of the parameters) to make networks more linearly connected (more merge-able via weight interpolation).  The authors show that a) increasing width reduces barriers (loss barriers require softmax recalibration), b) activations become linearly connected (remain more similar when linearly interpolated) with depth, dependent on two conditions (weak additivity and reciprocal orthogonality), and c) activations are lower rank with less \"overlap\" as width increases. Reducing weight decay (which increases rank) counteracts this effect.", "questions": "An interesting implication predicted by LEWC is that the exponential decay in activation magnitude as depth increases. This point isn't really touched on in the main text, and I would like to know if the exponential decay in the magnitude of activations from intermediate layers occurs as predicted:\n- What are the actual temperature values in fig 2 after calibration? Is LEWC predictive of the magnitude of these temperature values (i.e. does increasing depth make the scale of the logits smaller)?\n- Similarly, for fig 3 what does a plot of depth versus magnitude of the interpolated vector look like?\n\nIt isn't immediately obvious from the main text alone what the distinction is between the definition of reciprocal orthogonality and commutativity from Zhou et al. (2023) - in particular, what is the significance of the differences between the experiments for figure 8 in section 5.3 (empirically verifying reciprocal orthogonality) and figure 10 (empirically verifying commutativity doesn't hold)?\n\nCould the authors make a connection between LMC with increasing width, and the neural tangent kernel at the infinite width limit?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The work presents a novel and quite interesting finding, and backs it up with a predictive theory along with empirical evidence. The structure of the paper is well thought out and covers many of the gotchas and questions one may have going in - e.g. batch norm statistics are handled properly using REPAIR, and many efforts are made to distinguish reciprocal orthogonality from commutativity in Zhou et al. (2023). Overall there are many good things going on and I would like the authors to push a bit further beyond establishing that width-induced LMC exists.", "weaknesses": "I have two issues with this work. One issue is that the theory and experiments are somewhat detached as the theoretical setup and some predictions (depth particularly) don't have solid connections with the empirical evidence. Some of the definitions/conditions (e.g. low rank weights leading to LMC) feel arbitrary and accidental rather than clearly supported by intuition and evidence. The other issue is that I'm not sure what to conclude - can we predict when LMC will arise due to width alone? What are the implications (practical or otherwise) of the findings?\n\nCurrently, the empirical results cannot rule out the possibility that there is a lower floor on how small the barrier becomes with increased width. First, for the higher width scales, it is important to report the actual loss or accuracy barriers (such as in a table, zoomed-in plot, or plotting width vs max barrier) since \"LMC\" in the permutation literature (e.g. Ainsworth et al.) requires \"essentially no barrier\".\n\nAlong this line, I don't find the random permutations experiment (Figure 13) useful - in a sense, two randomly initialized and trained networks are already randomly permuted with respect to each other, since the probability of initializing $\\theta$ is equal to that of $\\pi(\\theta)$ for any permutation $\\pi$. Instead, I would like to know whether as width increases, the difference between barriers from a known good permutation and a random one goes to 0. For the known good permutation, one could use permutation-finding algorithms, or networks from the same initialization that are LMC (Frankle et al. 2020).\n\nI find the argument in 5.2 (low rank structure reduces overlap) somewhat unconvincing - first, the intuitive example is coordinate-aligned and I don't see how the same applies to arbitrary low-rank transformations which are not coordinate-aligned. It's possible that activations are coordinate-aligned in practice due to ReLU being a coordinate-aligned transform, which figure 6 supports. In general, I think a statement like \"width leads to sparser activations, which induces weak additivity\" would be stronger than the current \"low rank\" explanation.\n\nSmall problems with the presentation:\n- Missing citations for line 59: \"In addition, several studies have shown that widening ResNet-50 trained on the ImageNet dataset improves the test accuracy of the merged model when interpolating the weights of two trained models with permutation\"\n- The plot colors are hard to read for the different layer widths - maybe consider using a graduated palette (e.g. viridis)?\n- The \"standard deviation with respect to 0\" (defined in figu re 5) should be called the square root of the second (uncentered) moment\n- I suggest moving figure 5 to the appendix in favour of figure 9 - section 5.4 (reduced weight decay stops the effect) adds a much-needed ablation in my opinion. Actually, more ablations would really help establish that weak additivity and reciprocal orthogonality are necessary and sufficient conditions for LMC sans permutation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D4AXrq7UWD", "reviewer": "Reviewer_EvXf", "summary": "This paper explores the narrative around permutation symmetries and linear mode connectivity (LMC) from a new angle, finding that permutation symmetries are not necessary to achieve LMC for very wide models if one is allowed to recalibrate the temperature of the softmax (a much smaller modification). The paper then introduces layerwise exponentially weighted connectivity (LEWC), a nonlinear form of connectivity which is satisfied when the activations of the model decay exponentially as the layer index increases. The key observation is that low rank weights reduce the conflict between merged neurons and (relatively) large settings of weight decay promote low-rank weights.", "questions": "1. Do permutation methods like Git Re-Basin continue to work with typical choices of weight decay? If so, is the mechanism by which Git Re-Basin \"fixes\" the calibration of the model? Why doesn't LEWC continue to appear between the permuted models?\n2. Do you think the re-calibration step could further reduce the barriers for models merged with permutation methods in settings where LEWC does not arise organically (low weight decay)?\n3. Can you record the (midpoint) accuracies/losses of the models in numerical form? It it useful to see the exact numbers since differences in accuracy can be quite small while staying meaningful (for example). \n4. The larger setting of weight decay is important to establish the low-rank structure which allows the permutation-free merging to succeed. Given the merging does not succeed when the weight decay is set to a lower value of 1e-4, I think it would be interesting to see how the loss/accuracy barriers change as the weight decay decreases.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The key observation of the paper is very scientifically interesting and portrays linear mode connectivity and permutation symmetries in a new light. \n2. The paper also contains theoretical analysis justifying why LEWC can be expected to emerge for very wide models.\n   1. The theoretical analysis is based on two properties:\n   2. Weak Additivity for ReLU Activations (roughly, interpolating the inputs of ReLUs is the same as interpolating their outputs)\n   3. Reciprocal Orthogonality (roughly, each model's features live in the \"null space\" of the other)\n3. The above properties are tested in practice and found to emerge as the width increases. Both properties get strong as the width increases!", "weaknesses": "1. The main experiments use a slightly stronger than normal choice of weight decay (3e-3). I believe the typical values are roughly 1e-4 MNIST and 5e-4 for CIFAR, (although I am not familiar with the history of these choices). This is not a major weakness, but it does complicate the narrative since permutation-based methods still work when the weight decay is not so high.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d20Ks1MKVx", "reviewer": "Reviewer_3r8c", "summary": "This paper argues that sufficiently wide neural networks achieve linear mode connectivity without permutation alignment. While permutation-based methods require networks to have a minimum width, the authors claim that for a wide enough network the permutation search itself becomes unnecessary and provide insights on why it emerges. The authors extend layerwise linear feature connectivity (LLFC) by introducing layerwise exponentially weighted connectivity (LEWC), positing that the intermediate activations can be expressed as an exponentially weighted average of those of the individual models. They explain the emergence of LEWC with weak additivity of ReLU activations (same as LLFC) and reciprocal orthogonality.", "questions": "- Could you provide more analysis of the temperature parameter in Section 3.1? Is there a relationship between optimal temperature and network width/depth/alpha? How sensitive are the results to this choice? \n- I find the low-rank argument insightful. Given LMC is sensitive to optimization hyper-parameters (Altıntaş, et al. 2024) I would be interested to see a comparison between first and second-order methods in this line. Another experiment could be using no weight decay, since it is also a common practice in training Cifar-scale Resnets.\n- Have the authors investigated how the permutation-based methods affect the overlapping dimensions? Though not critical, this could be a good insight of the paper. I think an analysis in this line could support the claims in the paper.\n- I would be interested to see if the connectivity breaks gradually over layers and hence if there is a relationship between the width expansion factor and layer depth?\n- I think it would be useful if the authors commented on how normalization layers and residual connections interfere with the reciprocal orthogonality. The authors could experiment with this in the MLP architecture.\n- Even though the authors elaborate on LLFC (and commutativity not holding in their seeting) in the appendix I believe the paper could benefit from a broader discussion of the implications of LLFC vs LEWC as well as the emergence of LEWC in the spawning setting (Frankle et al, 2018). This could be an ablation of the paper.\n\nI find this line of research and the authors' contribution quite interesting and I am open to increasing my score if the authors can situate the implications of their work better within the existing literature.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The exponential derivation and the LEWC framework represent a meaningful extension to the layerwise linear feature connectivity, novel to the best of my knowledge.\n- The paper contributes to emerging literature questioning permutation alignment as necessary for mode connectivity, challenging a mainstream assumption in the field.\n- The paper provides a principled explanation for why LMC is more easily achieved in wider networks through the lens of reciprocal orthogonality, moving beyond the intuition that \"more width provides more permutations to search.\" This offers a different perspective on the role of overparameterization in mode connectivity.\n- The empirical framework and justification is sound.", "weaknesses": "-The paper makes it sound like the networks are widened post-hoc after training. The title and framing suggest a novel widening procedure (potentially post-hoc expansion of trained models, which would be very exciting), but the paper appears to simply experiment with networks that are already wide from initialization, this should be made clear. I believe post-hoc widening could be a very interesting addition to impose reciprocal orthogonality and low-rank structure.\n- The experiments inherit the limited scope of prior work and need expansion, some suggestions are included in the questions\n- I think the paper could benefit from an expanded discussion in relation to over-parameterized networks (e.g. Simsek et al, 2021).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gDvhA3WH3F", "reviewer": "Reviewer_RKhX", "summary": "The paper tackles the task of merging different checkpoints of the same architecture trained from scratch with different seeds. In particular, the paper investigates why linear mode connectivity increases just by increasing the width of the model, without the need for permutations. To explain this, they propose Layerwise Exponentially Weighted Connectivity (LEWC): the intermediate output of the merged model can be expressed as a weighted sum of those of the endpoint models, with the interpolation coefficients decaying exponentially with the widths. The paper derives two sufficient conditions for this phenomenon to arise: ReLU weak additivity and reciprocal orthogonality. The experiments then empirically confirm these ones to hold in several controlled settings, including MLPs, VGGs and ResNets over CIFAR10, MNIST and FMNIST.", "questions": "- To what extent are the observed results a consequence of extreme overparameterization and the smoothness of the loss landscape, rather than a new structural mechanism such as LEWC? \n- Do suggested experiments in Weakness 1 confirm LEWC? If these cannot be performed in the rebuttal time frame, what is the expected outcome?\n- Modern architectures have moved beyond ReLU. How does the treatment differ if one employs one more modern variant (e.g. SwiGLU)? \n- What are the practical implications? requiring strong weight decay and large width multipliers, does this actually inform or help any practical merging or federated learning pipeline?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is clearly written and easy to read. The exposition is well-structured, and the theoretical derivations are both intuitive and sound.\n- The problem is important, both in providing further understanding in loss landscape connectivity and carrying strong implications for model merging, editing and alignment.\n- The proposed theoretical framework is novel and interesting. Layerwise Exponentially Weighted Connectivity (LEWC) offers a way to generalize the “averaging as ensembling” point of view that is true for linear model to non-linear deep architectures. The sufficient conditions for LEWC to arise are formally rigorous and make intuitive sense.", "weaknesses": "- I have some reservations about the main claim of the paper. The observed permutation-free linear connectivity may stem from overparameterization-induced robustness rather than genuine connectivity. In very wide networks and easy tasks (e.g. CIFAR-10, MNIST), independent checkpoints can lie in flat regions of the loss landscape where large parameter perturbations (like naive weight averaging) don’t affect accuracy. In this perspective, the reported effect might be explained by functional redundancy and flatness rather than the what is stated in the paper. It would be interesting to see the following experiments to better understand if this is the case:\n    - Train seeds with constraints that violate the sufficient conditions for LEWC by setting  zero weight decay and spectral regularizers against low-rank. Measure ReLU weak additivity, reciprocal orthogonality and merge success. If merging still works despite not exhibiting relu weak additivity and reciprocal orthogonality then it doesn’t have to do with LEWC and it’s rather just functional redundancy.\n    - Compute the norm of the difference between A and B, then traverse a random direction *u* by that same distance and plot the loss barrier throughout. The loss should clearly increase in that direction, but if it’s still low at $\\lambda$ = 0.5 then we might be again observing functional redundancy.\n- Limited practical utility. The effect appears only for very wide models trained on simple datasets, under settings that are not those used in realistic large-scale systems. Moreover, the need for temperature calibration, strong weight decay, and substantial width multipliers means that the reported phenomenon does not affect practical model-merging pipelines.\n- Experimental evidence is limited to toy models and datasets. While this is somewhat common in prior literature, its applicability and theoretical impact would greatly benefit from considering more complex architectures such as ViTs. This could be done by following e.g. [1].\n\nOverall, I find the analysis and insights interesting and worth sharing, but requiring the proposed additional experiments to ensure we are not just observing spurious explanations for unknown, possibly trivial phenomena. \n\n[1] Theus, Alexander, et al. \"Generalized Linear Mode Connectivity for Transformers.\" NeurIPS 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)