Git Re-Basin: Merging Models modulo Permutation Symmetries
Authors: Samuel Ainsworth
Jonathan Hayase
Siddhartha Srinivasa
Venue: ICLR 2023 notable top 5% (2023)

Abstract: The success of deep learning is due in large part to our ability to solve certain massive non-convex optimization problems with relative ease. Though non-convex optimization is NP-hard, simple algorithms -- often variants of stochastic gradient descent -- exhibit surprising effectiveness in fitting large neural networks in practice. We argue that neural network loss landscapes often contain (nearly) a single basin after accounting for all possible permutation symmetries of hidden units a la Entezari et al. 2021. We introduce three algorithms to permute the units of one model to bring them into alignment with a reference model in order to merge the two models in weight space. This transformation produces a functionally equivalent set of weights that lie in an approximately convex basin near the reference model. Experimentally, we demonstrate the single basin phenomenon across a variety of model architectures and datasets, including the first (to our knowledge) demonstration of zero-barrier linear mode connectivity between independently trained ResNet models on CIFAR-10. Additionally, we identify intriguing phenomena relating model width and training time to mode connectivity. Finally, we discuss shortcomings of the linear mode connectivity hypothesis, including a counterexample to the single basin theory.

Published as a conference paper at ICLR 2023
GIT RE-BASIN: MERGING MODELS MODULO PERMU- TATION S SYMMETRIES
Samuel K. Ainsworth, Jonathan Hayase, Siddhartha Srinivasa Paul G. Allen School of Computer Science and Engineering
University of Washington {ska nswo, jhayase, s siddh ecs. wos.swashinggoo....oo edu
ABSTRACT
The success of deep learning due in large part to our ability to solve certain mas- sive non-convex optimization problems with relative ease. Though non-convex optimization is NP-hard, simple algorithms often variants of stochastic gradient descent exhibit surprising effectiveness in fitting large neural networks in prac- tice. We argue that neural network loss landscapes often contain (ncarly) a single basin after accounting for all possible permutation symmetries of hidden units a la Entezari et al.] (202T). We introduce three algorithms to permute the units of one model to bring them into alignment with reference model in order merge the two models in weight space. This transformation produces function- ally equivalent set of weights that lie in an approximately convex basin near the reference model. Experimentally. we demonstrate the single basin phenomenon across a variety of model architectures and datasets, including the first (to our knowledge) demonstration of zero-barrier linear mode connectivity between in- dependently trained ResNet models on CIFAR-10. Additionally, we investigate intriguing phenomena relating model width and training time to mode connectiv- ity. Finally, we discuss shortcomings of the linear mode connectivity hypothesis, including counterexample to the single basin theory.
INTRODUCTION
We investigate the unreasonable effectiveness of stochastic gradient descent (SGD) algorithms on the high-dimensional non-convex optimization problems of deep learning. In particular,
1. Why does SGD thrive in optimizing high-dimensional non-convex deep learning loss land- scapes despite being noticeably less robust in other non-convex optimization settings, like policy learning (Ainsworth et al., (202]) trajectory optimization (Kelly) 2017). and recom- mender systems (Kang et al. 2016)?
2. What all the local minima? When linearly interpolating between initialization and final trained weights, why does the loss smoothly and monotonically decrease (Goodfellow & Vinyals 2015 Frankle 20200 Lucas al. 2021 Vlaar & Frankle (001)}
3. How can two independently trained models with different random initializations and data batch orders inevitably achieve nearly identical performance? Furthermore, why do their training loss curves often look identical?
We posit that these phenomena point to the existence of some yet uncharacterized invariance(s) in the training dynamics causing independent training runs to exhibit similar characteristics. Hechtt- Nielsen 1990) noted the permutation symmetries hidden units in neural networks; briefly, one can swap any two units of a hidden layer in network and - assuming weights are adjusted accordingly
network functionality will not change. Recently, [Bnnton et al. (2021) demonstrated that SGD solutions form connected volume of low loss and Entezari et al. (22 conjectured that this volume is convex modulo permutation symmetrics.
Conjecture 1 (Permutation invariance, informal (Entezari et al. 202 )). Most SGD solutions belong to set whose elements can be permuted so that no barrier (as in Definition 2.2 exists on the linear interpolation between any two permuted elements.

---

Published as a conference paper at ICLR 2023
ARCHITECTURE
NUM. PERMUTATION SYMMETRIES
MLP (3 layers, 512 width)
10 A 3498
VGG16
10 A 35160
ResNet50
10 A 55109
Atoms in the observable universe
10 A 82
Table 1: Permutation symmetries of deep learning models upper estimate on the number of atoms in the known, observable universe. Deep learning loss landscapes contain incomprehen- sible amounts of geometric repetition.
We refer such solutions as being linearly mode connected (LMC) (Frnklle et al. 2020), an exten- sion of mode connectivity (Garipov et al.] 2018 Draxler et al.] 2018 .. true, Conjecture will both materially expand understanding of how SGD works in the context of deep learning and offer credible explanation for the preceding phenomena, in particular.
Contributions. In this paper, attempt to uncover what invariances may be responsible for the phenomena cited above and the unreasonable effectiveness of SGD in deep learning. We make the following contributions:
1. Matching methods. We propose three algorithms, grounded in concepts and techniques from combinatorial optimization, align the weights of two independently trained models. Where appropriate, we prove hardness results for these problems and propose approximation algorithms. Our fastest method identifies permutations mere seconds on current hardware.
2. Relationship to optimization algorithms. We demonstrate by means of counterexample that linear mode connectivity is an emergent property of training procedures, not of model architectures. We connect this result to prior work the implicit biases of SGD.
3. Experiments, including zero-barrier LMC for ResNets. Empirically, we explore the exis- tence of linear mode connectivity modulo permutation symmetries in experiments across MLPs, CNNs, and ResNets trained on MNIST, CIFAR-10, and CIFAR-100. We con- tribute the first-ever demonstration of zero-barrier LMC between two independently trained ResNets. We explore the relationship between LMC and model width as well as training time. Finally, we show evidence our methods' ability to combine models trained on inde- pendent datasets into merged model that outperforms both input models in terms of test loss (but not accuracy) and is no more expensive in compute or memory than either input model.
BACKGROUND
Although our methods can be applied arbitrary model architectures, we proceed with the multi- layer perceptron (MLP) for its ease of presentation (Bishop) 2007). Consider an L-layer MLP,
[(:; θ) 21+1, Zt++ a WW + bz), z1
where denotes an element-wise nonlinear activation function. Furthermore, consider loss, L(O), that measures the suitability of a particular set of weights θ towards some goal, c.g., fitting to a training dataset.
Central to our investigation is the phenomenon of permutation symmetries weight space. Given O, apply some permutation to the output features of any intermediate layer, f, of the model, denoted by a permutation matrix P E St
*++1 P Pzi+1 P Po b,) P o oPWW Pb:)
for a, an clement-wise operator. It follows that long as reorder the input weights of layer é according to PT, we will have functionally equivalent model. To be precise, if we define O to be identical θ with the exception of
W/ = PWe b Pbe, W441 = WT
We denote the set of all d d permutation matrices isomorphic the symmetric group as Sz. to the possible chagrin of pure mathematicians.
2

---

Published as a conference paper at ICLR 2023
then the two models functionally equivalent: f(z; () f(z θ²) for all inputs This implies that for any trained weights θ, there is an entire equivalence class of functionally equivalent weight assignments, not just one such θ, and convergence to any specific element of this equivalence class, as opposed to any others, is determined only by random seed. We denote a functionality- preserving permutation of weights л (())
Consider the task of reconciling the weights of Iwo, independently trained models, and B, with weights θ A and B, respectively, such that we can linearly interpolate between them. We assume that models A and B were trained with equivalent architectures but different ran- dom initializations, data orders, and potentially different hyperparameters
Git Re-Basin
क ((O)
datasets, as well. Our central question is: Given O and Ø, can we identify some и such that when linearly in- terpolating between a and iT ()))), all inter- mediate models enjoy performance similar to A and O₂?
a
B
Figure I: Git Re-Basin merges models by tele- porting solutions into single basin. OB per- muted into aunctioalll-equiialle T(OB) so that lies the same basin O
We base any claims of loss landscape convex- ity the usual definition of multi-dimensional convexity in terms of one-dimensional convex- ity per
Definition 2.1 (Convexity).. A function f RD - convex if every mi-dimensional slice is convex, i.e., for all x, E RP, the function g(A) = f((1 - A)z + Xy) is convex in X.
Due to Definition 2.1 it suffices to show that arbitrary one-dimensional slices of a function are con- vex in order to reason about the convexity of complex, high-dimensional functions. practice, we rarely observe perfect convexity but instead hope to approximate it closely as possible. Following Frankle et al. (2020); Entezari et al. (2021); Draxler et al. (2018); Garipov et al. (2018) and others, measure approximations convexity via "barriers."
Definition 2.2 (Loss barrier (Frankle al. 2020 Given points O Og such that L(OA) x B) the loss barrier is defined as maxxe C((1 - A) + X0 (C((A)) + L(OB)).
Loss barriers are non-negative, with zero indicating an interpolation of flat or positive curvature.
PERMUTATION SELECTION METHODS
We introduce three methods of matching units between model and model B. Further, we present an extension simultaneously merging multiple models in Appendix A.10 and an appealing but failed method in Appendix A.1I
3.1 MATCHING ACTIVATIONS
Following the classic Hebbian mantra, "[neural network units] that fire together, wire gether" (Hebb) 2005), we consider associating units across two models by performing regression between their activations. Matching activations between models is compelling since it captures the intuitive notion that two models must learn similar features to accomplish the same task (Li et al.) 2016). Provided activations for each model, we aim to associate each unit in A with a unit stands to reason that linear relationship may exist between the activations of the two models. We fit this into the regression framework by constraining ordinary least squares (OLS) to solutions in the set of permutation matrices, S4. For activations of the f'th layer, let ( (B) E Rdxa denote the dl-im. activations for all n training data points in models A and B, respectively. Then,
Pt arg min PES.
Z (A)
PZ
arg max (P, Z (B)) PeSa
(I)
i=1
3

---

Published as a conference paper at ICLR 2023
where (A, B) A., denotes the Frobenius inner product between real-valued matrices A and B. Conveniently, eq. (T] constitutes "linear assignment problem" (LAP) (Bertsekas) 1998) for which efficient, practical algorithms are known. Having solved this assignment problem on each layer, we can then permute the weights of model B to match model A as closely as possible
W (.W( (B) PEL1. b/= P.b
for each layer E, producing weights O' with activations that align as closely possible with O
Computationally, this entire process is relatively lightweight: the ( and Z matrices can be computed in a single pass over the training dataset, and, practice, a full run through the training dataset may be unnecessary. Solving eq. (1) is possible due to well esllbbssablished polynomial-time al- gorithms for solving the linear assignment problem (Kuhn) 2010 Jonker & Volgenant 1987 Crouse, 2016). Also, conveniently, the activation matching each layer is independent of the matching every other layer, resulting in a separable and straightforward optimization problem; this advantage will not be enjoyed by the following methods.
Dispensing with regression, one could similarly associate units by matching against a matrix of cross-correlation coefficients in place of Z(4) (( We observed correlation matching to work equally well but found OLS regression matching to be more principled and easier to implement.
Activation matching has previously been studied for model merging in Tatro et al. (2020); Singh & Jaggi 2020); Li ct al. (2016) albeit not from the perspective of OLS regression.
3.2 MATCHING WEIGHTS
Instead of associating units by their activations, we could alternatively inspect the weights of the model itself. Consider the first layer weights, W1; each of Wi corresponds single feature. If two such rows were equal, they would compute exactly the same feature (ignoring bias terms for the time being). And, if [W(A').. la. ze [w:( 1b.. it stands to reason that units and j should be associated, Extending this idea to every layer, we are inspired to pursue the optimization
arg min A))(O - (e)( = arg max vec(OA) vec(T(Og)).
We can re-express this in terms of the full weights,
arg max (W(4), (.w( + (W!4), Paw! B) P. P (W(A) w( P (2) x=(P.)
resulting in another matching problem. We term this formulation the "sum of bilinear assignments problem" (SOBLAP). Unfortunately, this matching problem is thornier than the classic linear as- sigmment matching problem presented in eq. TT.. Unlike LAP, we are interested in permuting both the rows and columns of w to match w( which fundamentally differs from permuting only rows or only columns. We formalize this difficulty as follows.
Lemma 1. The of bilinear assignments problem (SOBLAP) is NP-hard and admits polynomial-time constant-factor approximation scheme for > 2.
Lemma D contrasts starkly with classical LAP, for which polynomial-time algorithms are known.
Undeterred, we propose a approximation algorithm for SOBLAP. Looking at single P. while holding the others fixed, observe that the problem can be reduced to classic LAP,
arg max (W(4), P.W. ( PLINF + (W(A)
6+1 P PEt1 (B) E+1
P )
Pt
arg (Pe, W (A) (w B (W P (.)..
P.
This leads to a convenient coordinate descent algorithm: go through each layer and greedily select its best Pt. Repeat until convergence. We present this in Algorithm
Although we present Algorithm D in terms of an MLP without bias terms, in practice our imple- mentation handle the weights of models of nearly arbitrary architectures, including bias terms, residual connections, convolutional layers, attention mechanisms, and so forth. We propose an ex- tension of Algorithm to merging more than models at time in Appendix A.
4

---

Published as a conference paper at ICLR 2023
Inageial widi)
Yedre
Inh
3.8-
3.44
1.50
Figure 2: Linear mode connectivity is possible after permuting. Loss landscapes when interpo- lating between models trained MNIST, CIFAR-IO, and ImageNet. In all cases we can signifi- cantly improve over na've interpolation. Straight-through estimator matching performs best but is very computationally expensive. Weight and activation matching perform similarly, although weight matching is orders of magnitude faster and does not rely on the input data distribution. We hypoth- esize that the ImageNet barrier could be reduced by increasing the model width as in Section 5.3
Algorithm 1: PERMITATTONCOORDNWATEEEEEER
Given: Model weights θ - (w(A) ..... w/a1) and θB {w10) ..... W/ (B)
Result: permutation π = (PI..... Pt-11 } of On such that vec(O vec(T(( is approximately maximized.
Initialize: Pi t I....,P--1tt! 1 repeat
for E RANDOMPEMUTTTION[] 1)) 1)do do
P. SOLVELAP (W(1)) (e + (W(Al)T T We-1
end
until convergence
Lemma 2. Algorithm terminates.
Our experiments showed this algorithm to be fast in terms of both iterations necessary for conver- gence and wall-clock time, generally on the order of seconds to few minutes.
Unlike the activation matching method presented Section 3.1 weight matching ignores the data distribution entirely. Ignoring the input data distribution and therefore the loss landscape handicaps weight matching but allows it to much faster. We therefore anticipate its potential application in fields such as finetuning IDevlin et al.) 2019; Wortsman et al. 2202bb a). federated learning (McMa- han et al. 2017 Konečný et al. 2016a 6a b), and model patching (Matena & Raffel 2021 Sung et al., 2021 Raffel 2021). In practice, found weight matching to be surprisingly competitive with data-aware methods. Section (isionnSsstucca studies this trade-off
3.3 LEARNING PERMUTATIONS WITH A STRAIGHT-THROUGH ESTIMATOR
Inspired by the success of traighh-through estimators (STEs) in other discrete optimization prob- lems (Bengio al., 2013] Kusupati al., 2021 Rastegari et al., 2016 Courbariaux & Bengio 2016), we attempt here to "learn" the ideal permutation of weights π T Specifically, our goal is to optimize
min c (e proj (6x)) @n
proj(O) A arg max vec(O) vec(T(@B)), (3)
where θB denotes approximation of (Og),, allowing to i optimize However, eq. (3) involves inconvenient non con-differentiable projection operations, proj(-). complicating the op- timization. We overcome this via "straight-througe" estimator: parameterize the problem in terms of set of weights θg x(O)). In the forward pass, project Ô to the closest realizable (O)). In the backwards pass, we then switch back to the unrestricted weights Ô g- In this way,
5

---

Published as a conference paper at ICLR 2023
MNIST
CIFAR-10
0.6
Train
0.5
Test
barrier
Loss barrier
0.4
0.20
0.3
0.15
0.10
0.05
0.0
0.00
10
15
20
25
10
15
20
25
Epoch
Epoch
Figure 3: Linear mode connectivity challenging at initialization. We show loss barriers per training time for MLPs trained on MNIST (left) and CIFAR- 10 (right). Loss interpolation plots are inlaid to highlight results in initial and later epochs. LMC manifests gradually throughout training. We hypothesize that the variance in CIFAR-10 training is higher due to our MLP architecture being under-powered relative to the dataset. (Y-axis scales differ each inlaid plot.)
are guaranteed stay true to the projection constraints in evaluating the loss but can still compute usable gradients at our current parameters, θ
Conveniently, we can re- re-purpose Algorithm| I solve proj(OB). Furthermore, we found that ini- tializing θB O performed better than random initialization. This is to be expected immediately at initialization since the initial matching will be equivalent to the weight matching method of Sec- tion 3.1 However, it is not immediately clear why these solutions continue to outperform a random initialization asymptotically.
Unlike the aforementioned methods, Algorithm 2 Jattempts to explicitly "learn" the best permutation using conventional training loop. By initializing to the weight matching solution of Section 3.2 and leveraging the data distribution as in Section 3.1] seeks to offer best-of-both worlds solution. However, this comes at very steep computational cost relative to the other two methods.
4 A COUNTEREXAMPLE TO UNIVERSAL LINEAR MODE CONNECTIVITY
In this section we argue that common optimization algorithms, especially SGD and its relatives, are implicitly biased towards solutions admitting linear mode connectivity. In particular, we demon- strate by way of counterexample that adversarial, non-SGD solutions exist in loss landscapes such that no permutation of units results linear mode connectivity, We present this counterexam- ple in complete detail in Appendix A.6
The existence of adversarial basins suggests that our ability to find LMC between independently trained models is thanks to inherent biases in optimization methods. We emphasize that this coun- terexample does not contradict Conjecture T rather, it illustrates the importance of the conjecture's restriction SGD solutions (Entezari et al. 2021). Characterizing the precise mechanism by which these solutions are biased towards LMC could be an exciting avenue for future work.
We also note that there are invariances beyond permutation symmetries: It is possible to move features between layers, re-scale layers, and so forth. Prior works noted the feature/layer associa- tion (Nguyen et al. (021) and re-scaling invariances (Ainsworth et al. 2018). The importance of these other symmetries and their interplay with optimization algorithms remains unclear.
5 EXPERIMENTS
Our base methodology is to separately train two models, and B, starting from different random initializations and with different random batch orders, resulting in trained weights O and Og.
NNte again that projecting according to inner product distance is equivalent to projecting according to the La distance when parameterizing the estimator based the B endpoint. We also experimented with learning the midpoint directly, é ~ ( + r(@B)), in which case the L2 and inner product projections diverge. In testing all possible variations, we found that optimizing the B endpoint had slight advantage, but all possible variations performed admirably.

---

Published as a conference paper at ICLR 2023
VGG-16
ResNet20
1.2
1.50
Train
1.0
1.25
Test
barrier
Loss barsier
0.8
1.00
0.6
0.4
0.2
0.0
0.00
16x
16.
1x
2
4x
1x
2x
B Width multiplter
4x
16x
32x
Width multiplier
Figure 4: Wider models exhibit better linear mode connectivity. Training convolutional and ResNet architectures on CIFAR-10, we ablate their width and visualize loss barriers after weight matching. Notably, we achieve zero-barrier linear mode connectivity between ResNet models, the first such demonstration.
respectively. We then evaluate slices through the loss landscape, L((1 - e) + (T(O))) for
A E [0, 1], where is selected according to the methods presented in Section 3 Ideally, we seek a completely flat or even convex one-dimensional slice. As discussed in Section 2 the ability to exhibit this behavior for arbitrary A OB empirically suggests that the loss landscape contains only single basin modulo permutation symmetries.
We remark that failure to find a π such that linear mode connectivity holds cannot rule out the existence of a satisfactory permutation. Given the astronomical number of permutation symmetries, Conjecture iis essentially impossible to disprove for any realistically wide model architecture.
5.1 Loss LANDSCAPES BEFORE AND AFTER MATCHING
We present results for models trained MNIST (eCunn et al. 1998). CIFAR-10 Krizhevsky 2009). and ImageNet (Deng et al. 2009) in Figure 2 Naive interpolation (T) θn) sub- stantially degrades performance when interpolating. On the other hand, the methods introduced in Section
3 can achieve much better barriers. We achieve zero-barrier linear mode connectivity on MNIST with all three methods, although activation matching performs just slightly less favorably than weight matching and straight-through estimator (STE) matching. We especially note that the test loss landscape becomes convex after applying our weight matching and STE permutations! In other words, our interpolation actually yields merged model that outperforms both models A and B. We elaborate on this phenomenon Section 5.4 and Appendix(AAI)
On ImageNet we fall short of zero- barrier connections, although we do see a 67% decrease in barrier relative to naive interpolation. As we demonstrate in Section (5.3) we can achieve zero-barrier LMC on CIFAR-10 with large ResNet models. Therefore, we hypothesize that the presence of LMC depends on the model having sufficient capacity (esp. width) to capture the complexity of the input data distribution, and that ImageNet results could be improved by expanding model width.
STE matching, the most expensive method, produces the best solutions. Somewhat surprising, how- is that the gap between STE and the other two methods is relatively small. In particular, it is remarkable how well Algorithm performs without access to the input data at all. We found that weight matching offered compelling balance between computational cost and performance: It runs in mere seconds (on current hardware) and produces high-quality solutions.
[
5.2 ONSET OF MODE CONNECTIVITY
Given the results of Section 5.1 it may be tempting to conclude that the entirety of weight space contains only single basin modulo permutation symmetries. However, we found that linear mode connectivity is an emergent property of training, and we were unable to uncover it early training.
We explore the emergence of LMC in Figure 3 Concurrent our work, Benzing et al. (2022) showed that LMC at initialization is possible using permutation found at the end of training.
We also experimented with spherical linear interpolation ("slerp") and found it to perform slightly better than lincar interpolation in some cases; however, the difference was not sufficiently significant to warrant diverging from the pre-existing literature.
7

---

Published as a conference paper at ICLR 2023
Note that the final inlaid interpolation plot in FigureBiright demonstrates important shortcoming of the loss barrier metric, i.c., the interpolation includes points with lower loss than either of the two models. However, the loss barrier is still positive due to non-negativity, as mentioned in Section 2
5.3 EFFECT OF MODEL WIDTH
Conventional wisdom maintains that wider architectures are easier to optimize (Jacot et al. 2018 Lee al., 2019). We now investigate whether they are also easier to linearly mode connect. We train VGG-16 (Simonyan & Zisserman 2015) and ResNet20 (He et al. 2016) architectures of varying widths on the CIFAR-10 dataset. Results are presented in Figure
A clear relationship emerges between model width and linear mode connectivity, measured by the loss barrier between solutions. Although 1x-sized models did not seem to exhibit linear mode connectivity, we found that larger width models decreased loss barriers all the way zero. In FigureFrightt right), we show what is to our knowledge the premiere demonstration of zero cero-barrier linear mode connectivity between two large ResNet models trained on non-trivial dataset.
We highlight that relatively thin models do not seem to obey linear mode connectivity yet still exhibit similarities in training dynamics. This suggests that either permutation selection methods failing to find satisfactory permutations on thinner models or that some form of invariance other than permutation symmetries must be play in the thin model regime.
5.4 MODEL PATCHING, SPLIT DATA TRAINING, AND IMPROVED CALIBRATION
Inspired by work finetuning Wortsman et al. 2022a), model patching (Singh & Jaggi 2020]]Rafte (02T)) and federated learn- ing (McMahan et al. 2017 Konečný et al.] 2016alb), we study whether is possible to synergistically merge the weights of two mod- els trained on disjoint datasets. Consider, for example, an organization with multiple (pos- sibly biased) datasets separated for regulatory (e.g., GDPR) or privacy (e.g., on-device data) considerations. Models can be trained on each dataset individually, but training in aggregate is not feasible. Can we combine separately trained models that the merged model per- forms well on the entirety of the data?
Split data training
Naive weight interp. Ensemblling
4
matching
Tost loss
Combined training
Model Dataset
Model Datasel B
Figure 5: Models trained on disjoint datasets can be merged constructively. Algorithm makes it possible for two ResNet models trained on disjoint, biased subsets of CIFAR-I00 to be merged in weight space such that their combina- tion outperforms both input models in terms of test loss on the combined dataset.
To address this question, split the CIFAR- 100 dataset (Krizhevsky] (209) into two dis- joint subsets: dataset A, containing 20% exam- ples labelled 0-49 and 80% labelled 50-99, and dataset B, vice versa. ResNet20 models and B were trained on their corresponding datasets. Privacy requirements mandate that we utilize a data-agnostic algorithm like Algorithm|1 [T] Figure the result of merging the two models with weight matching. For comparison, we benchmark naïve weight interpolation, ensembling of the model logits, and full-data training.
As expected, merging separately trained models did not match the performance of an omniscient model trained on the full dataset or an ensemble of the two models with twice the number of effec- tive weights. On the other hand, we did manage merge the two models in weight space, achieving an interpolated model that outperforms both input models terms of test loss while using half the memory and compute required for ensembling. Furthermore, the merged model's probability esti- mates are better calibrated than either the input models as demonstrated in Figure TT Accuracy results are presented in Figure TO Algorithm also vastly outperformed na've interpolation, the status quo for model combination in federated learning and distributed training.
'Unfortunatlly 8x width VGG-16 16 training was unattainable since it exhausted GPU memory available hardware at the time of writing.
8

---

Published as a conference paper at ICLR 2023
6 RELATED WORK
(Linear) mode connectivity, Garipov et al. (0018); Draxler et al. (2018) : Freeman & Bruna (2017)
showed that different solutions in the neural network loss landscape could be connected by paths of near-constant loss, which Garipov et al. (2018) coined "mode connectivity." Tatro et al. (2020) ex-
plored non-linear mode connectivity modulo permutation symmetries. Frankle et al. (2020) demon- strated connection between linear mode connectivity and the lottery ticket hypothesis. Juneja et al.
(2022) demonstrated that LMC does not always hold, even when fine-tuning. [Hecht-Nielsen 1990);
Chen et al. (1993) noted the existence of permutation symmetries, and [Brea et (2019) implicated them as source of saddle points in the loss landscape. Recently, the prescient work of [Entezari al. (2021) conjectured that SGD solutions could be linear mode connected modulo permutation symmetries and offered experiments buttressing this claim. Unlike previous works on LMC we ac- complish cero-barrier paths between two independently-triin on the order of seconds.
models with algorithm that runs
Loss landscapes and training dynamics. Li et al. (2016); Yosinski et al. (2014) investigated whether independently trained networks learn similar features, and to what extent they transfer. Jiang al.] (2021) argued that independently trained networks meaningfully differ in the features they learn in certain scenarios. Zhang et al. (2019) studied the relative importance of layers. Ben- lon et al. (202I) argued that SGD solutions form connected volume of low loss. Pittorino et al. (022)) proposed a toroidal topology of solutions and a set of algorithms for symmetry removal. On the theoretical front, Kawaguchi (2016) proved that deep linear networks contain no local minima. Boursier et al. 2202);; Chizat & Bach][2118 Mei et al. (2018) characterized the training dynamics of one-hidden layer networks, proving that they converge to zero loss. Godfrey et al. (2022); [Sim-] isek et al. (2021) investigated the algebraic structure of symmetries in neural networks and how this structure manifests in loss landscape geometry.
Federated learning and model merging. McMahan al. (2117): Konečný et al. 20 b) in- troduced the concept of "federated learning. i.e., learning split across across multiple devices and datasets. Wang et al. (2200) proposed an exciting federated learning method in which model averag- ing is done after permuting units. Unlike this work, they merged smaller "child" models into larger "main" model, and did so with layer-wise algorithm that does not support residual connections or normalization layers. Raffel ); Matena & Raffel (2ff)))(202)) Sung et al. (202T) conceptualized the study of "model patching." i.e., the idea that models should be easy modify and submit changes to. [lharco et al. (2022) investigated model patching for the fine-tuning of open-vocabulary vision models. AAsmmor Gashler (2015) first explored the use of matching algorithms for the alignment of network units. [Singh & Jaggi (2020) proposed merging models by soft-aligning associations weights, inspired by optimal transport. Liu al. (2022a): Uriot Izzo| (2020) further explored merging models taking possible permutations into account. Wortsman et al. (2022a) demonstrated state-of-the-art ImageNet performance by averaging the weights of many fine-tuned models.
7 DISCUSSION AND FUTURE WORK
We explore the role of permutation symmetries the linear mode connectivity of SGD solutions. We present three algorithms to canonicalize independent neural network weights in order to make the loss landscape between them as flat as possible. In contrast to prior work, we lincarly mode connect large ResNet models with no barrier in seconds to minutes. Despite presenting successes across multiple architectures and datasets, linear mode connectivity between thin models remains elusive. Therefore, we conjecture that permutation symmetries are necessary piece, though not complete picture, of the fundamental invariances at play in neural network training dynamics. In particular, we hypothesize that linear, possibly non-permutation, relationships connect the layer- wise activations between models trained by SGD. In the infinite width limit, there exist satisfactory linear relationships that are also permutations.
An expanded theory and empirical exploration of other invariances such as cross-layer scaling or general linear relationships between activations - presents an intriguing avenue for future work. UI- timately, we anticipate that a lucid understanding of loss landscape geometry will not only advance the theory of deep learning but will also promote the development of better optimization, federated learning, and ensembling techniques.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
Merging models raises interesting ethical and technical questions about the resulting models. Do they inherit the same biases as their input models? Are rare examples forgotten when merging? Is it possible to gerrymander a subset of the dataset by splitting its elements across many shards?
Deployment of any form of model merging ought to be paired with thorough auditing of the resulting model. investigating in particular whether the merged model is representative of the entirety of the data distribution.
REPRODUCIBILITY STATEMENT
Our code is open sourced at wttpttts:/gggiiiia com/ saauela/gii reeeeeebaaae basin, Our experi- mental logs and downloadable model checkpoints are fully open source attpps:: / /wwndba ai/ skainswo git wete//iitttee--eeeeeeeeeeeeeeeee
ACKNOWLEDGMENTS
We are grateful to Vivek Ramanujan, Mitchell Wortsman, Aditya Kusupati, Rahim Entezari, Jason Yosinski, Krishna Pillutla, Ofrr Press, Matt Wallingford, Tim Detmers, Raghav Somani, Gabriel Ilharco, Ludwig Schmidt, Sewoong Oh, and Kevin Jamieson for enlightening discussions. Thank you John Thickstun and Sandy Kaplan for their thoughtful review of earlier draft of this work and to Ofir Press and Tim Dettmers for their potent advice on framing and communicating this work. This work was (partially) funded by the National Science Foundation NRI (#2132848) & CHS (#2007011), DARPA RACER, the Office of Naval Research, Honda Research Institute, and Amazon. This work supported in part by Microsoft and NSF grants DMS-213401 and CCF- 2019844 as part of NSF Institute for Foundations of Machine Learning (IFML).
10

---

Published as a conference paper at ICLR 2023
REFERENCES
Samuel K. Ainsworth, Nicholas J. Foti, Adrian K. C. Lee, and Emily B. Fox. oi-vae: Output interpretable vaes for nonlinear group factor analysis. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Stockhelmsmüüssan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. vBC/ wtt.w/attsottaaaa.mmo html
Machine Learning, ICML 2018,
PMLR, 2018. URL Utt mlr press/
Samuel K. Ainsworth, Kendall Lowrey, John Thickstun, Zaid Harchaoui, and Siddhartha S. Srini- vasa. Faster policy learning with continuous-tinn gradients. In Ali Jadbabaie, John Lygeros, George J. Pappas. Pablo Parrilo, Benjamin Recht, Claire Tomlin, and Melanie N. Zeilinger (eds.). Proceedings of the 3rd Annual Conference on Learning for Dynamics and Control, LADC 2021, 7-8 June 2021, Virtual Event, Switzerland, volume 1 144 of Proceedings of Machine Learn- ing Research, pp. 1054-100000 1067. PMLR, 2021. URL http::/ppooceddiiii wnw.innn..mm.....mmmm.mmm....... mtr.ssss...rrs////// 11.4//annswort2211
html
Stephen C. Ashmore and Michael S. Gashler. A method for finding similarity between multi-layer perceptrons by forward bipartite alignment. In 2015 International Joins Conference on Neural Networks, IJCNN 2015, Killarney, Ireland, July 12-17, 2015, pp. 1-7. IEEE, 2015. doi: 10.1 20 77807666. URL https::::: /oi. org /10. 1109/ 1119/JJCN.... 2O1.:/NNNN.................. 7280769
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrry E. Hinton. Layer normalization.
CoRR,
abs/ abs/1607.06450, 2016. URL http:://paxiv...// wttp://www../ga////iaaaaaa/ abs/1607,...///////// (000//660700400 06450
Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308.3432. 2013. URL
http: //tt:v//rxiv... org/ aaaaaass/111 3432
Gregory W. Benton, Wesley J. Maddox, Sanae Lotfi, and Andrew Gordon Wilson. Loss surface simplexes for mode connecting volumes and fast ensembling, In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18- 24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 769 779. PMLR, 2021. BRLhttp:: hattp://proceediingg mlrr wttpppsse/////eettttroooo html
Frederik Benzing, Simon Schug, Robert Meier, Johannes von Oswald, Yassir Akram, Nicolas Zuc- chet, Laurence Aitchison, and Angelika Steger. Random initialisations performing above chance and how to find them, 2022. Utl/tttss://axxiv.o wttpxxii.orr//b///2////a http://bb/2220...1.12 07509
D.P. Bertsekas. Network Optimization: Continuouy and Discrete Methods. Athena scientific optimization and computation series. Athena Scientific, 1998. ISBN 9788865290279. URL
https://books. google www.:/bookkss.aaaEEAA......
Christopher M. Bishop. Pattern recognition and machine learning, 5th Edition. Information science and statistics. Springer, 2007. ISBN 9780387310732. URL, ftttttt///// /www. wwwldcat..............o......... org/
0clc/71008143
Mathieu Blondel, Olivier Teboul, Quentin Berthet, and Josip Djolonga. Fast differentiable sort- ing and ranking. In Proceedings of the 37th International Conference Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 950-959, PMLR, 2020. URL http:////ppr mlr. ptt../prrsss///////// blonde 20a html
Etienne Boursier, Loucas Pillaud-Vivien, and Nicolas Flammarion. Gradient flow dynamies of shal- low relu networks for square loss and orthogonal inputs. CoRR, 000p//2206000999, 2022, doi: 10.448550/axxx122266.99999 URLhttps: : /ooi 005../0.................... 2206. 00939
Johanni Brea, Berfin Simsek, Bernd Illing, and Wulfram Gerstner. Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss land- scape. CoRR, abs/1 907.0077000 0000022911..1 1, 2019. URL

---

Published as a conference paper at ICLR 2023
E. Cela. The Quadratic Assignment Problem: Theory and Algorithms. Combinatorial Optrmmizaiiniiiiiiiiiiii http////c.....oooiie.ooooo///ooo
Springer US, 2013. ISBN 9781475727876. URL ott.////...../.../////////o.ooo.
[d==0OGCAARQBAJ
An Mei Chen, Haw-minn Lu, and Robert Hecht-Nielsen. On the geometry of feedforward neural network error surfaces. Neural Compur., 5(6):910 927, 1993. doi: 00.1162/neco.19933355 1993.5.6.910. 1993.5.6..
URL //doips:/do..... htt/////.or//111////////////////
Lénaïc Chizat and Francis R. Bach. On the global convergence of gradient descent for over-parameterized models using optimal transport. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Ad- vances in Neural Information Processing Systems 31: Annual Conference on Neural Informa- tion Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 3040-3050, 2018. URL https:://proceeeinnnge neurips.... wwwpaaurpippppppppee/e.eeea /c/paperr201B/hastt/ Abstract 0tl.c55866c35544000772ee33e at html
Matthieu Courbariaux and Yoshua Bengio. Binarynet: Training deep neural networks with weights and activations constrained to +1 or -1. CoRR, abs/1602.02830, 2016. URL http://arxiv:
akg/abb/1602... 02830
David Frederic Crouse. On implementing 2d rectangular assignment algorithms. IEEE Trans. Aerosp. Electron. Syst. 1264))167- 2016. doi: 10.1109/TAES.2016 140952. URL
https:://ol.org// 11..:11199/TAA. 2016. 140952
Marco Cuturi, Olivier Teboul, and Jean Phillipe Vert. Differentiable ranks and sorting using optimal transport. CoRR, abs/1905.1 1885, 2019. URL puttp://arxvvy. org/ 1905. 1 1885
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miani, Florida, USA, pp. 248-255. IEEE URL httph:tp/////mm htttps::/aii.ogg//// org/0/
Computer Society, 2009, doi: (L00//CVP.2009955066 11C9/CVPR. 2 00 5206848
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACI-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pp. 4171- 4186. Association for Computational Linguistics, 2019. doi: 10.18653/v1 1/133/11/111///// 1423. URL https://aoi.org// http://0..16633/11///
1423
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred A. Hamprecht. Essentially no bar- riers neural network energy landscape. In Jennifer G. Dy and Andreas Krause (eds.), Proceed- ings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Re- search, pp. 1308-1311 PMLR, 2018. URL http: http::/prroceedingg ml press/v8// draxlerr1 aater8aahhmmm html
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. CoRR, abs/2110.06296 2021. URL
https://erxivv wtwpviissr///os.o.o 22110006220
Fajwel Fogel, Rodolphe Jenatton, Francis R. Bach, and Alexandre d' Aspremont. Convex relaxations for permutation problems. SIAM J. Matrix Anal. Appl., 36(4):1465 -460-1488,888, 2015. doi: 10.1137/
130947362. URL httpt:::///oi.oooo wwt.:orggg.....
Jonathan Frankle. Revisiting "qualitatively characterizing neural network optimization problems". CoRR, abs/2012.06898, 2020. URL htt.tttss//axxi...... http://aaxii..ogga//////.22.///. abs///a...22/2///////.//a/..a. 06898
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In Proceedings of the 37th International Conference Machine Learning. ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 3259-322320 3269. PMLR, 2020. URL htLhttp: /ttp::///ppoceeengg.... m wtt.ppess/w19/fffffepe e20a. html
12

---

Published as a conference paper at ICLR 2023
C. Daniel Freeman and Joan Bruna. Topology and geometry of half-rectified network optimiza- tion. In 5th International Conference on Learning Representations. ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL hetps: //openreviex wwt. wtw.pmmmmm.dKkKKWcge.c
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P. Vetrov, and Andrew Gordon Wil- son. Loss surfaces, mode connectivity, and fast ensembling of dnns. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Anmual Conference on Neural Infor- mation Processing Systems 2018, NewrlPS 2018, December 3-8, 2018, Montréal, Canada, pp. 8803 8812, 2018. URL
neurips CC www./ppppe/..//////ecc .tml
Charles Godfrey, Davis Brown, Tegan Emerson, and Henry Kvinge. On the symmetries of deep learning models and their internal representations. CoRR, abs/2205, 14258, 2022. doi: 10.48550/ 1rXiv.2205.14258 8 http://arxxv.222.1.14..
Ian J. Goodfellow and Oriol Vinyals. Qualitatively characterizing neural network optimization prob- lems. In Yoshua Bengio and Yann LeCun (eds.). 3rd International Conference Learning Rep- resentations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL httt:://ar/et
abs////////////aaaa 2112111214 6544
Aditya Grover, Eric Wang, Aaron Zweig, and Stefano Ermon. Stochastic optimization of sort- ing networks via continuous relaxations. In 7th International Conference on Learning Repre- sentations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL
https:// net.eeww.eeeeeeeeeeeeeeeeeeee forum ? Fid H1eSS3CcKK
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp- 770-778. IEEE Computer Society, 2016. doi: 10.1109/CVPR.22066.99. URL /tt. wtt.://///....////............
1109/CVPR. 2016. 90
Donald Olding Hebb. The organization of behavior: A neuropsychological theory. Psychology Press, 2005.
Robert Hecht-Nielsen. On the algebraic structure of feedforward network weight spaces. 1990.
Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannanch Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by in- doi: 10.485500rxxw.22000.5555 URL
terpolating weights. CoRR, abs/2208 05592, 2022.
https:://or www.oo.g..............ooo....... 48550/ arxiv 2208. 05592
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis R. Bach and David M. Blei (eds.), Proceedings the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 448 456. JMLR.org, 2015. URL mlr wwwpsrssssseee wtt.///fe/......... html
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wil- son. Averaging weights leads to wider optima and better generalization. In Amir Globerson and Ricardo Silva (eds.), Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial In- telligence, UAI 2018, Monterey, California, USA, August 6-10, 2018, pp. 376-885, AUAI Press, 2018.
Arthur Jacot, Clément Hongler, and Franck Gabriel. Neural tangent kemnl: Convergence and generalization in neural networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicold Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Anmual Conference on Neural Information Processing Systems 2018, NewrIPS 2018. December 3-8, 2018. Montréal, Canada, pp. 8580-8589, 2018 URL /tL[:tttpp::/procedaaao http://prooeeddn...............s hctp:/sc.cc/aapp/1////aaae.. 5a4be fa34e62bb8..mmmm (tw.e222688eeeec6111111222226600 Abstttaaaaatttttttta.....
13

---

Published as a conference paper at ICLR 2023
Yiding Jiang, Vaishnavh Nagarajan, Christina Back, and J. Zico Kolter. Assessing generalization of SGD via disagreement. CoRR, 2ess2206611799, 2021. httpp:://aaxx..oootttt /bs/
2176.
Roy Jonker and A. Volgenant. A shortest augmenting path algorithm for dense and sparse lin- car assignment problems. Computing, 38(4):325-340, 1987. doi: 10.1007/BF02288700. URL
Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur. REPAIR:
renormalizing permuted activations for interpolation repair. CoRR, abs/2211.08403, 2022. doi:
10.48550/ 1.08403. wTw URL/httpss:/doo.orr///0 http:///o................/////..
08403
Jeevesh Juneja, Rachit Bansal, Kyunghyun Cho, João Sedoc, and Naomi Saphra. Linear connectiviyyy reveals generalization strategies. CoRR, abs/2205. 2022. doi: 10.48500/rxxiv22005111111
URL http::tt::///t //doi org. 48100/.85555/88
2241. 12411
Zhao Kang, Chong Peng, and Qiang Cheng. Top-n recommender system via matrix completion. In Dale Schuurmans and Michael Wellman (eds.), Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA, pp. 179-185. AAAI Press, 2016. URL http::ttt//ww aaai org/ www./o/g//////o//////oooo.oooo// iit//nndxxxx////// pbtpAAAT/AAAAT///pPPpS view/11824
Kenji Kawaguchi. Deep learning without poor local minima. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29: Annual Conference on Neu- ral Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp- 2016. URL https::/ mew.: neurips poc//apprr/2016//1 E2IE399255C11ccc2511112323323332 Abssteereeed--atttttataa.a
Abstract .html
Matthew Kelly. introduction to trajectory optimization: How to do your own direct collocation.
SIAM Rev., 59(4):849-90) 2017. doi: 10. 10.1137//6MM1062569. URL
1137/1 6M1062569
Jakub Konečný, H. Brendan McMahan, Daniel Ramage, and Peter Richtárik. Federated optimiza- tion: Distributed machine learning for on-device intelligence. CoRR, abs/1610.02527, 2016a. 02527
URL http::////axxi org/
Jakub Konečný, H. Brendan McMahan, Felix X. Yu, Peter Richtárik, Ananda Theertha Suresh, and
Dave Bacon. Federated learning: Strategies for improving communication efficiency. CoRR, abs/1610.05492, 2016b. URL
05492
Tjalling C. Koopmans and Martin Beckmann. Assignment problems and the location of economic activities. Econometrica, 25(1):53-76, 1957. ISSN 00129682, 14680262. httphtt:://www
Sstor.org/sttabe/199774
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Rohith Kuditipudi, Xiang Wang, Holden Lee, Yi Zhang, Zhiyuan Li, Wei Hu, Rong Ge, and Sanjeev Arora. Explaining landscape connectivity of low-cost solutions for multilayer nets. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Ro- man Garnett (cds.), Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurIPS 2019, December 8- 14, 2019, Vancou- ver, BC, Canada, pp. 14574-14583, 2019. URL Utt.:atttss//pprceeedii
cc/ wwwrips. ccw.ccss/ccccccc...c...
Harold W. Kuhn. The hungarian method for the assignment problem. In Michael Jünger, Thomas M.
Liebling, Denis Naddef, George L. Nemhauser, William R. Pulleyblank, Gerhard Reinelt, Gio- vanni Rinaldi, and Laurence A. Wolsey (eds.), 50 Years of Integer Programming 1958-2008 doi: 10.1007/
From the Early Years to the State-of-the-Art pp. 29 47. Springer, 2010.
978-3-540-68279-0) 2. URL U http::ttt////t. //ooi org 10. 1007 978 3- 540 682798877899
14

---

Published as a conference paper at ICLR 2023
Aditya Kusupati, Matthew Wallingford, Vivek Ramanujan, Raghav Somani, Jae Sung Park, Krishna Pillutla, Prateek Jain, Sham M. Kakade, and Ali Farhadi. LLC: accurate, multi- purpose learnt now-dimensional binary codes. In Marc' Aurelio Ranzato, Alina Beygelz- imer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference Neural Informa- tion Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 23900- 23913, 2021. URL
neurips......................s.s hcc/pappr//c2///// html
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. IEEE, 86(11))227-2533) 1998. doi: 10.1109/5.726791. URL
https://doi www.oooggg........ 11.99/......... 726791
Jachoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as lin- ear models under gradient descent. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelz- imer, Florence d' d'Alché-Buc, Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neu- ral Information Processing Systems 32: Annual Conference on Neural Information Pro- cexsing Systems 2019, NewrlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp- neuripss wttpcc//ppppa Abstract
3570-8581, 2019. URL aastaassssssbbbbaaaaaatattttt. html
http://d//sssaeerirrre/////////p wwtpppppe/////////eeece /
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John E. Hopcroft. Convergent learning: Do different neural networks learn the same representations? In Yoshua Bengio and Yann LeCun (eds.). 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http: /ttt::://axiii org/ abs/ 151
07543
Chang Liu, Chenfei Lou, Runzhong Wang. Alan Yuhan Xi, Li Shen, and Junchi Yan. Deep neural network fusion via graph matching with applications model ensemble and federated learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 13857-13869. PMLR, 2022a. URL
ml www..ppres//e v 626 1i222ku22kkkk html
Zhuang Liu, Hanzi Mao, Chao-Yus Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 11966-119766 IEEE, 2022b. doi: 10. 0D09CCCPRR52688.2222.1111 URL https:: http::///dooooggg//// 1109/ CWW.C22PR22222 2022.
01167
Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. https::::: penttps://operreee http:///ppeeeeeeeeeeeeeeee net/ net/ ftt//ooumm?cc htt///ooummidsdsee (oorummiddskk$95ccx
James Lucas, Juhan Bae, Michael R. Zhang, Stanislay Fort, Richard S. Zemel, and Roger B. Grosse On monotonic linear interpolation of neural network parameters. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18- 24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 7168- 7179. PMLR, 2021. URL ptst:://ppoceedingg. mlr. wwwpppsssp....... http////////uullaaaaaaaaaaaaaaaa html
Wesley J. Maddox, Pavel Izmailov, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wil- son. A simple baseline for bayesian uncertainty in deep learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Informa- fion Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 13132-13 2019. ht / neuripe http:///ww....iooooooo.oop////// wttpp//pccccpppppcppe./epp haaahh719991ee223332222223333332
html
15

---

Published as a conference paper at ICLR 2023
Konstantin Makarychev, Rajsekar Manokaran, and Maxim Sviridenko. Maximum quadratic signment problem: Reduction from maximum label cover and Ip pp-ddeed approximation algo- rithm. ACM Trans. Algorithms, 10(4):18:1-18.15 2014. doi: 10.1145/262672.. URLhttps:
//doi.org//l http///111111/261111 1145/2629672
Michael Matena and Colin Raffel. Merging models with wesher-weighted averaging. CoRR, 09832
abs/211 1.09822, 2021. URL htt.hhtp//////
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication-efficie learning of deep networks from decentralized data. In Aarti Singh and Xiaojin (Jerry) Zhu (eds.), Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS 2017, 20-22 April 2017, Fort Lauderdale, FL, USA, vol- ume 54 of Proceedings of Machine Learning Research, pp. 1273-1282. PMLR, 2017. URL www.nnngsssm.....mmmmmm.e.m..eem mwr. press v544 mwmamammmaaaacccaaaaamaaamaaa 7a.html
http://proceedinng
Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two-layers neural networks. CoRR, 0oos/804400661.. 2018. URL
1804.06561
Gonzalo E. Mena, David Belanger, Scott W. Linderman, and Jasper Snoek. Learning latent permu- tations with gumbel-sinkhom networks. In 6th International Conference on Learning Represen- tations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceed- ings. OpenReview.net, 2018. BRLLLtttss///peeeeett
net. forum? BB- Ow
Thao Nguyen, Maithra Raghu, and Simon Korbbltth. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, net. forum?id
May 3-7, 2021. OpenReview.net, 2021. URL htttp:://openret
K KJNcAkY8CY4)
Felix Petersen, Christian Borgelt, Hilde Kuehne, and Oliver Deussen. Differentiable sorting networks for scalable sorting and ranking supervision. CoRR, abs/2105.04019, 2021. URL
wttp:/rg//bbs////a 2215. 04019
Felix Petersen, Christian Borgelt, Hilde Kuehne, and Oliver Deussen. Monotonic differentiable sorting networks. CoRR, abs/2203.096300, 2022. doi: 10448550/arx..2222336666 URL https:
(rdbi.orgg/00 48550/arXiv. 2203. 09630
Fabrizio Pittorino, Antonio Ferraro, Gabriele Perugini, Christoph Feinauer, Carlo Baldassi, and Riccardo Zecchina. Deep networks on toroids: Removing symmetries reveals the structure of flat regions in the landscape geometry. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (cds.), International Conference Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learming Research, pp. 17759- 17781. PMLR, 2022. URL battps://ppt http://///opceepeeeeeeeee mmrr. press/v1622 wttsss/w11///itttroooo22.. html
Sebastian Prillo and Julian Eisenschlos. Softsort: A continuous relaxation for the argsort operator. In Proceedings of the 37th International Conference on Machine Learning. ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 7793-7802. PMLR, 2020, URL
mlr. prress/vll 111199/prllo2000 html
Colin Raffel.
A call to
build
models
like
we
build open-
source
https://colinraffet www.fffll.c.mm.lloo.o. a cacllll 1--calltto- build models like we buil build-op open sourcceuuceeeee software. html
software.
2021. Accessed: 2022-06-17.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.). Computer Vision - ECCV 2016 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV. volume 9908 of Lecture Notes in Computer Science pp. 525-542, Springer, 2016. doi: 10. 000000777788331-4666060 https://doo .tw.i..rgggooooo 1007 3-
32. URL
16

---

Published as a conference paper at ICLR 2023
Sartaj Sahni and Teofilo F. Gonzalez. P-complete approximation problems. J. ACM. 23(3):555- 565, 1976. doi: 10.1145/321158.522977. URL B2L975
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun (eds.). 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Truck Proceed- ings, 2015. URL
Berfin Simsek, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerst- ner, and Johanni Brea. Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances. In Marina Meila and Tong Zhang (eds.), Proceedings the 38th International Conference on Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual Event, vol- ume 139 of Proceedings of Machine Learning Research, pp. 9722-9732. PMLR, 2021. URL sttp://smmsa2a..mmm 2tm.mmmaammmmmmmmmmmmmmmmmmmmm
http://proceedinn
mmlr.
Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florins Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL htttps://proceediing negrs.nneripp.. ips. cc/paaper/222/hhat. Abstract.
1b201166111880006eeee233333 atsrreeesssebbltt--ttttttaatttt .tmll
Yi-Lin Sung, Varun Nair, and Colin Raffel. Training neural networks with fixed sparse masks. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann Dauphin, Percy Liang, and Jennifer Wort- man Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Confer- ence on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 24193 24205, 2021. URL httpp:://proceednngg hew.eeeeeeiinnee..n............. nwwrips...p.... cc/ paper/ 2021/hash/ 2t22://2222211111100888888331312
N. Joseph Tatro, Pun-Yu Chen, Payel Das, Igor Melnyk, Prasanna Sattigeri, and Rongjie Lai. Optimizing mode connectivity via neuron aligmment. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances Neural Information Processing Systems 33: Annual Conference on Neu- ral Information Processing Syxtems 2020, NeurlPS 2020, December 6-12, 2020, vir- tual, 2020. URL
neuripp cc/paper/202// 2200///sh//2 Abstract html
Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky. Instance normalization: The missing ingredient for fast stylization. CoRR, abs/ htss/1607.00022 2016. URL http://arxiv..zz org
0as/1607.08022
Thomas Uriot and Dario Izzo. Safe crossover of neural networks through neuron alignment. In
Carlos Artemio Coello Coello (ed.), GECCO '20: Genetic and Evolutionary Computation Con- ference, Cancún Mexico, July 8-12, 2020, pp. 435 435 443, ACM, 2020. doi: 10.1145/3377930. 3390197. URL httph: org 11e.:/...111113131 0000799993333333 3390197
Tiffany Vlaar and Jonathan Frankle. What can linear interpollation of neural network loss landscapes tell us? CoRR, 16004, 2021. URL https: wttpxxxxxvvvrog.//.........o..oo abs /20// 16004
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno A. Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL htttp:://openrewe
net/ for///orrmm/// i dd=uxi 3bZLkr3c
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris S. Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. In 8th International Conference on Learning Repre- sentations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URI. https::/openrewt
www.:nnttto.tteee.e forummid-dsaaroo USFSS
Mitchell Wortsman, Maxwell Horton, Carlos Guestrin, Ali Farhadi, and Mohammad Rastegari. Learning neural network subspaces. In Marina Meila and Tong Zhang (eds.), Proceedings of
17

---

Published as a conference paper at ICLR 2023
the 38th International Conference Machine Learning, ICML 2021. 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 11217-11227. PMLR, 2021. BBLLAttp.//prooeednnss..t wtt.a//pppeee//////oottttttaa
html
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without creasing inference time. Proceedings of the 39th International Conference on Machine Learning, ICMI, 2022, 2022a.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannanch Hajishirzi. Ali Farhadi, Hongseok Namkoong, al. Robust linettnnig of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7959-7971, 2022b.
Yuxin Wu and Kaiming He. Group normalization. 755. 2020. doi: 10. 10.1007/s1 1233-019-111-- 019:--11---- 111263-019-011:0
Int. J. Comput. Vis., 128(3):742-
URL Dettss://ac..ogg/10
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger (eds.). Advances in Neural Information Processing Systems 27: Anmual Conference Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pp. 3328, 2014. URL pttpp::/proceeddiiooooo ngs neurips cc/
paper/ httpr///14/aahha///e
Chiyuan Zhang, Samy Bengio, and Yoram Singer. Are all layers created equal?
CoRR,
abs/1902.01999 2019, URL htthtt::// arxiv. org/ abs abtp://.s.1110000000 01996
18

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 KNOWN FAILURE MODES
We emphasize that none of the techniques presented in this paper are silver bullets. Here we list the failure cases that the authors are presently aware of,
1. Models of insufficient width
2. Models in the initial stages of training
3. VGGs on MNIST
4. MNIST MLPs trained with SGD and too low of a learning rate, or Adam and too high of a learning rate
5. ConvNeXt architectures (Liu al. 2022b), which have surprisingly few permutation sym- metries due to extensive use of depth-wise convolutions
Furthermore, we believe other failure modes certainly exist but have yet to discovered.
We are excited by the prospect of future work investigating these failure modes and improving our understanding of when and why model merging modulo permutation symmetries feasible.
A.2 EXTENDED RELATED WORK
Non-linear mode connectivity. flourishing set of literature exists studying non-linear mode con- nectivity, including but not limited to Garipov et al. (2018): Draxier et al. (2018): Kuditipudi et al. (219)). This insightful line work is inspirational to our own, however we take a strictly linear ap- proach to mode connectivity as in Frankle et al. (2020); Juneja et al. (2022). Restricting ourselves to linear trajectories comes with the advantage having direct implications for single-basin theory. However, it comes at the cost of more challenging, discrete optimization problem. In particular, we found that in contrast to non- non-linear mode connectivity linear mode connectivity becomes drastically harder with smaller width models. Note additionally that most pre-existing mode nectivity work does not account for permutation symmetries of weight space, a linchpin element of our work. A notable exception to this trend can be found Tatro et al. (2020).
To summaaize: Freeman & Bruna (017) introduced the notion of mode connectivity and proved that loss landscapes for single-hidden layer ReLU models contain only single basin in the infinite width limit. Garipov et al. (2018) and Draxler et al. (2018) concurrently demonstrated that simple zero-barrier curves can be learned connect the optima in weight space, thus reshaping our un- derstanding of practical loss landscape geometries. Kuditipudi et al.] (2019) proposes a theoretical explanation for the mode connectivity phenomenon. Benton et al. (2021) extends mode connectivity from one-dimensional paths to entire manifolds of low-loss, and show that these manifolds can be leveraged for ootte-of-thcaat Bayesian ensembling of models.
Relationship with Tatro et al. (2020). The impact of permutation symmetries the non-linear mode connectivity models is considered in Tatro et al. (2020). particular, they independently but use it in conjunction with learned non-linear mode connecting curves. In contrast, we show that linear mode connectivity can be achieved without the need for learning non-linear paths between the aligned weights. Our derivation of Section 3.1 from the principle of least-squares regression is novel, to the best of our knowledge.
propose an algorithm more-or-less equivalent to Section 3. but
Relationship with Singh & Jaggi (2020). Singh & Jaggi (020) studies model merging with "soft matchings" between units from the perspective of optimal transport. We emphasize the following commonalitis/differeenes with their work:
We focus linear mode connectivity modulo permutation symmetries and its implications for a single- basin theory. On the other hand, /Singh & Jaggi (2020) emphasizes "soft" (ie., non permutation) matching of units via optimal transport.
single-basin
The activation matching method of [Singh & Jaggi (2020) reduces that of Section 3.1 when the optimal transport regularization term is set to zero and the unit "importance" values are uniform across all units on all layers.
19

---

Published as a conference paper at ICLR 2023
Our weight matching and traiggh-through estimator methods solve for alignment across all layers jointly. In contrast, Singh & Jaggi (2020) executes greedy, single-pass pass matching looking only at weight information from the immediately previous layer when selecting
permutations. [Singh & Jaggi (202)) suggests jointly solving for alignments as an avenue
for future work.
The "wts" method of SSnghh Jaggi (2020) not run on models including bias terms, skip works with models of connections, or normalization layers. In contrast, our Algorithm [
nearly arbitrary architecture.
outperforms the "wts" method of |Singh & Jaggi 2020). See Appendix A.7 for more information.
Our weight matching method (Algorithm I
Singh Jaggi 2020) introduces a method for merging multiple models simultaneously. but only demonstrates results on at most 8 models at a time and performs continued training after merging- In contrast, our Algorithm| [ has been shown to work with as many as 32 models at a time and does not require continued training after merging. Our analysis of the calibration of the resulting merged models has no parallel in Singhh & Jaggi (2020).
Relationship with Entezari et al. (2021). Entezari et al. (022I) introduces the single-basin conjec- ture and provides the following evidence towards it:
Entezari et al. (2021) provides statistical test which fails to detect difference in bar- rier statistics between independently trained models and random permutations of the same model (Fig. 5 of Entezari et al. (2021)) Our work provides stronger support for the con- jecture in that we give methods that can directly "unscramble" these permutations, proving that LMC can be found (Figures 4 and
Entezari et al. (202T)'s experimental protocol does not provide evidence for linear mode connectivity. Rather, their experimental results suggest that barriers resulting from inde- pendent training look like the barriers resulting from random permutations. But this result is consistent with a world in which all solutions have barriers between them both be- tween members of the same permutation equivalence class and between solutions in sepa- rate equivalence classes! In other words, there may still exist multiple equivalence classes of solutions. In contrast, we provide concrete evidence for a single-basin theory by devel- oping algorithms that directly place independent solutions into the same basin (Figure [.
Although EEtezarii et al. (202)> 's conjecture is an important intellectual ancestor to our work, their demonstration of linear mode connectivity is limited to single hidden-layer MLP on MNIST (Fig. 2 of Entezari et al. (202I)). However, this result for single hidden- layer MLP models is preceded by Freeman & Bruna (2017): Uriot & Izzo (2220). On the other hand, we focus on larger models and datasets that are more closely aligned with models used in practice at the time of writing.
Entezari et al. (2021) proposes a simulated annealing algorithm that yields modest reduc- tions in barrier between independently trained models, yet requires multiple days to run.
On the other hand, our weight matching algorithm (Algorithm[I [ completely removes barri- ers between models for more challenging models and datasets (Figures (nndeess(add (5). and runs in seconds (Appendix A.5). Moreover, our weight matching method does not require access to the training data, enabling its potential application in domains like federated learning and distributed training.
In short, the work Entezari et al. (2021) first proposed the "single-bass basin" conjecture. Our work is the first (to the best of our knowledge) to demonstrate that linear mode connectivity can be achieved between large models independently trained on challenging datasets.
Differentiating through permutations. Akin to differentiable permutation learning, many prior works have studied differentiable sorting (Grover et al. 2019 Prillo & Eisenschlos) 2020 Cuturi al. 2019: Petersen et al. 2022; [021] Mena et al. 2018). Blondel et al. (2020) studied differ- entiable sorting and ranking with asymptotics that correspond to their non- non-differentiable versions. Fogel et al. (2015) explored recovering the linear orderings of items based pairwise informa- tion, another form of permutation optimization. [engiio et al. (2013) introduced the straight- estimator for differentiating through discrete projections that we utilize in Section 3.3
20

---

Published as a conference paper at ICLR 2023
A.3 EXPERIMENTAL DETAILS
A.3.1 MULTI-LAYER PERCEPTRON MODELS
In these experiments we utilized networks with hidden layers of 512 units each. ReLU activations were used between layers and no normalization was performed. Optimization was done with Adam and learning rate of le 3.
A.3.2 VGG- 16 AND RESNET MODELS ON CIFAR DATASETS
We utilized the VGG-16 16 architecture of Simonyan & Zisserman 2015) with the exception that we used LayerNorm normalization in place of BatchNorm. Similarly used the ResNet20 architecture He et al. (016)) but with LayerNorms in place of BatchNorms.
The following data augmentation performed during training
Random resizes of the image between 0.8x-1.2x
Random 32x32 pixel crops
Random horizontal flips
Random rotations between =:30"
Optimization was done with SGD with momentum (momentum set to 0.9). A weight decay reg- ularization term of 5e 4 applied. A single cosine decay schedule with linear warm-up was used. Learning rates were initialized at le - 6 and linearly increased to le 1 over the span of an epoch. After that point a single cosine decay schedule (Loshchilov Hutter 2017) was used for the remainder of training.
A.3.3 RESNET5O MODELS ON IMAGENET-IK
In this experiment we utilized pre-trained ResNet50 model available for download online, and one trained ourselves. These were standard ResNet50 models, including the use of BatchNorm. In line with prior work (Izmailov et al. 2018, Wortsman et al.] 202T Maddox et al.) 2019, Wang et al. 2021), recalculate BatchNorm statistics after performing weight interpolation. After our initial publication, the recalculation of BatchNorm statistics was suggested to us by the authors of Jordan al. (2022).
A.4
THE RELATIONSHIP BETWEEN PERMUTATION MATCHING AND NORMAL.IZATION LAYERS
In this section discuss the impact that different types of common normalization layers have on the feasibility of model merging.
BatchNorm (loff & Szegedy 2015) generally breaks after interpolating between weights
due to the so-called "variance collapse" problem (Jordan et al.) (022). Therefore, we rec- ommend the recalculation of batch statisties after merging models (Izmailov et al. 2018
Wortsman et al. 2021 Maddox et al. 2019 Wang et al. (021).
LayerNorm (Ba et al. 2016) is invariant to permutations of units and found that archi- tectures with LayerNorm can be merged without issue.
InstanceNorm (Ulynnov al., 2016) also places no restrictions on unit order, and in prin- ciple does not present any issues, although we have not run any experiments with it.
GroupNorm (W & He 2020) relies on unit indexes to organize units into groups, and therefore is not invariant to permutations of units. In principle, permutation alignment methods would not work on architectures with GroupNorm, though we have not tested this.
A.5 ADDITIONAL INFORMATION ON ALGORITHM 1
On currently available hardware (p3.2xlarge AWS instance with an NVIDIA V100 GPU), we ob- served the following timing results with Algorithm
21

---

Published as a conference paper at ICLR 2023
permutations
Shop lavity
0.20
Loos
0.15
Idontity
0.10
Swap layer
Swap layer
Swap both
performance
0.00
Model A
Model
Seog
Figure 6: counterexample to universal LMC. There exist models such that no possible per- mutation of weights allows for linear mode connectivity. Left: performance of all possible linear interpolations between the two models. Right: visualization of the prediction functions f(x) through each linear sweep. Each row corresponds to of the four possible permutations, and each column corresponds to a value of X, the linear interpolant. The existence such cases suggests that linear mode connectivity is artifact of SGD.
Data
1.0
0.5
X2
0.0
-05
-1.0
XX
1.0
0.5
0.0
0.5
1.0
X1
Figure 7: The counterexample classification problem data.
1. MLP (3 layers, 512 units each): seconds
2. ResNet50 (1x width): 33 seconds
3. ResNet20 (32x width): 194 seconds
In addition, we tested the ability of Algorithm recover known, randomly selected permutation. In a handful of experiments we found that Algorithm [ random permutation in just 3-4 of passes over the layers.
was able to exactly recover the known,
A.6 COUNTEREXAMPLA DETAILS
Consider a simple 2-dimensional classification task. Our data points are drawn x Uniform[[11,1]] and y 1x, <0 and xy>0. Figure 7 provides visualization of a sample of such data.
We utilize MLP architecture consisting of two hidden layers, with two units each, and ReLU nonlinearities. Consider two weight assignments that both achieve a perfect fit to the data:
0
0
fa(x) = [-1 -
-1
+
+
(4)
1
0
0
0
fz(x) = [-1 -
0
+
(1)
(5)
-1
We predict a positive label when f(x) > 0 and negative label otherwise.
22

---

Published as a conference paper at ICLR 2023
Intuitively, these networks are organized such that each layer makes classification whether æ1 or 22 > O. In model A, the first layer tests whether 22 > O, and the second layer tests whether 21 0, whereas in model B the order is reversed. With bit of algebra, it is possible to that both Sa and fn achieve perfect performance. However, no possible permutation of units results in linear mode connectivity between SA and fB. We visualize all possible permutations in Figure
We claim that this example, and the underlying trick, are simple enough to be embedded into larger models. For example, this could trivially be extended ResNets where different subsets of layers could be set to identity functions.
As diseussed in Section 4 the existence of adversarial basins in the loss landscape has interesting consequences for our understanding of loss landscape geometry. In particular, we argue that this implies that common optimization algorithms are conveniently biased towards solutions that admit LMC. However, the precise connection between optimization algorithms and linear mode connec- tivity remains unclear.
This counterexample does not constitute contradiction of the conjecture in Entezari et al. [2021) To be more precise, Conjecture S, of parameter space such that every pair of elements in S can be linearly mode connected (after some permutation of units), and that with high probability SGD solutions are contained in S. The example presented in this section does not contradict Entezari et al. (2021)'s s conjecture, but instead illustrates that the restriction to SGD solutions is a "load-bearing" element the conjecture.
Entezari et al. (202T) proposes that there exists some subset,
A.7 ON THE FAILURES OF GREEDY UNI-DIRECTIONAL MATCHING
In contrast to prior work (Pittorino et al. 2022 Singh & Jaggi 2020 Wang et al. 2020), we eschew greedy uni-directional, single-pass matching between models. Instead we derive a weight matching algorithm from principled, yet computationally infeasible optimization problem. In contrast to prior work, our method can be viewed as "bi-directional": it selects unit associations based on weights in all relevant layers, not just in the immediately previous layer. In this section, we describe benefits of our holistic approach, including an example problem showing the failure modes of greedy uni-directional matching. We find that matching across all layers simultaneously allows our weight matching algorithm (Algorithm I to exploit units' relationships with downstream weights in way that greedy umi-directional matching cannot.
Concretely, greedy uni-directional matching begins at the first layer and computes matching, Pi, considering only (A) (( After matching, P1 is applied throughout the remainder of the network. Then, we proceed through the layers in order repeating this process.
We compare Algorithm to greedy uni-directional matching experimentally in Appendix A.771 and present a theoretical counterexample that illustrates the advantages of our method in Ap- pendix A.7.2
A.7.1 EXPERIMENTAL COMPARISON
In order to further evaluate our performance relative to prior work, and Pittorino et al. (2022): Singh Jaggi (2020) in particular, we explored experimental comparisons with VGGTT models trained on CIFAR-10 10 and on ResNet50 models trained on ImageNet.
VGG11 models trained on CIFAR-10. Following an exact reproduction of experiment from Table 1 Sfnghh & Jaggi (2020), we merged the model weights released along with their paper. We present We found that our weight matching method outperforms the "wts" method of Singh & Jaggi (2020) in both implementation speed and model performance when reproducing their experiment with their trained model weights.
results in Table
ResNet50 models trained on ImageNet. We applied OT-Fusion ("wtts") to the ImageNet exper- iment that we consider in Section [.1 We present results Figure g We found the OT-Fusion method resulted in models with 1.38% top-1 accuracy ImageNet, only marginally improving over na've averaging. On the other hand, we achieve 51.01%.
NarchNorm statistics were recalculated after interpolation for all methods shown.
23

---

Published as a conference paper at ICLR 2023
Method
Test accuracy (f)
Run-time (L)
OT-Fusion (Singh & Jaggi 2020)
$5.98%
2.86s
Weight matching (ours)
86.57 %
0.64s
Table 2: VGGLI/CIFAR-10 performance relative Singh & Jaggi (2020). We found that our
weight matching method outperforms the "wts" method of Singh & Jaggi (2020) in both implemen- tation speed and model performance when reproducing one of their experiments with their published model weights. Our implementation is 4.5x faster, and produces solution with better model per- formance.
ImageNet, ResNet50 (1x width)
ImageNet, ResNet50 (1x width)
Top-1 Acpuracy
Top-5 Accuracy
Natve
60
OT-Pusion
Weight matching (ours)
40
20
20-
0-
Model
Model
ModelA
Model
A
Figure 8: ResNet50/mmaeeNet performance relative to Singh & Jaggi (020). We found the OT-Fusion method resulted models with 1.38% top-1 accuracy on ImageNet, only marginally improving over naive averaging. On the other hand, we achieve 51.01%.
Although a number of factors may be responsible for the difference in performance between weight matching and OT-Fusion, we found the number of aligmment passes made over the network layers have a substantial impact. OT-Fusion is inherently limited to a single pass over the layers. On the other hand, we are not limited to any specific number of optimization passes and instead continue until convergence (convergence guaranteed by Lemma2). For comparison, our weight matching algorithm is artificially handicapped single pass over the layers, achieve similarly low 7% top-1 accuracy.
A.7.2 AN EXAMPLE FAILURE CASE
Consider two networks, and B, with the objective that they capture the identity function f(x) x,
0
fe [1
0
+
0 0
Sea (z) [0
0
1++)
where 0 is some negligible constant. It be seen that these reduce to fea (x) and feg (x) = (1 e)x.
When aligning these models there are possible opportunities for permutation, (P,, Pz). The permuted model then has the form
1+t)=
(x) = (lo IPP))
P
P P
(8)
Now, aligning with greedy uni-directional matching will result in the alignment gud (Pi
I. P2 I}. On the other hand, our weight matching method (Algorithm I
results in 7wm
0 P2 =
[0 1
p
0 regardless of the algorithm's execution order.
24

---

Published as a conference paper at ICLR 2023
MINIST XLP
CIPAR 10, MLP
CIPAR 10, ResNet20 (32X WOTTL
103
100
60m
Aucuracc
250
80-
40
Toat Text
07-
M600A
Modet
Figure 9: Top-1 1 accuracy results for the MNIST and CIFAR-10 10 models of Figure
CIFAR-100, Split data training
CIFAR-100. Split data training
100
TTpp1 acouracy
60-
Top-5 accuracy
90-
80
70-
Model Dataset
Model Dataset B
Model Dataset
Model Dataset
A
A
Figure 10: Accuracy results for the CIFAR-100 split data experiment.
Interpolating these matched models at À
0.5, we have
0.5
0
1
[ [1(0 [0.5
0.5]
0
(1
f2
1+
x
= (0.5 O(e)+
(9)
0
1
c/2 e/2
(410) = [1 0]
(1 + c/2)a
(10)
c/2
Here we can see that the greedy uni diiidrrectional matching (Equation (9) results a merged model that fails to represent the input, edentity-function models. On the other hand, weight matching algorithm (Algorithm[T Equation (1O)) produces a merged model that accurately reflects both the input models, and even improves performance over the B model.
A.8
AUXILIARY PLOTS
A.9
STRAIGHT-THROUGH ESTIMATOR DETAILS
See Algorithm 2
for a complete description of the straight-through estimator algorithm.
10
MERGING MANY MODELS
We propose Algorithm to merge the weights of more than two models at time.
Following argument similar to Lemma 2 it can be seen that Algorithm Srrnithhmmttrinnnnng Eterminates.
In our limited testing, we found that this algorithm converges quickly to solutions that extrapolate better than individual models and results in merged model with better probability estimate calibra-
5 We note this example can be extended to the more conventional presentation, including non-linear activation functions, by inserting ReLU activations in each layer and considering in the positive domain I The positive domain restriction can also be lifted by instead considering the task of absolute value estimation and adding layers [1 1 to the beginning of the network.
25

---

Published as a conference paper at ICLR 2023
CIFAR-100 Split Datasets, Calibration
Train
Test
1.00
1.00
Perfect calibration
Model A
True probability
0.75
Model B
"True prrobabiit
0.75
Naïve merging
Model ensemble
0.50
Weight matching
0.50
0.25
0.25
0.00
0.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Predicted probability
Predicted probability
Figure 11: Merging CIFAR-100 split data models results in superior probability calibration. Although merged model is not competitive in terms of top-1 accuracy in the CIFAR- 100 split data experiment, we find that it has far better calibrated probability estimates than either of the input models. In addition, achieve calibration results par with model ensembling while requiring 2x less memory and compute.
ImageNet, ResNet50 (1x width)
ImageNet, ResNet50 (1x width)
80
80
Top-1 Accuracy
60
Top-5 Accuracy
60
40
40
20
Model A
Model 8
Model
Model B
Figure 12: Accuracy results for merged ResNet50 (1x width) models on ImageNet.
Algorithm 2: Straight-through estimator training
Given:
Model weights A, θB, and learning rate n permutation π of Og such that (( + T(OB))) is approximately minimized.
Result:
Initialize: θB +- θ
repeat
(On) + ( using Algorithm
Evaluate the loss of the midpoint, L( T - ((0))))
Evaluate the gradient, VL, using θ in place of ((0p) in the backwards pass.
Update parameters, θg θ V.
until convergence
26

---

Published as a conference paper at ICLR 2023
Algorithm 3: MERGEMANY
Given: Model weights I,,... NN
Result:
A merged set of parameters θ.
repeat for i RANIDOPPEMUTATINNN............., do
O' - Nーーーーン2 E 6
PRRUUTTTDDYCOODDIMIIDDSSSCTTT
: + ((;)
end
until convergence
return
N 5Z
Train Loss
Train Acc.
Test Loss
Test Acc.
Seed
0.0000
1.0000
0.1153
0.9856
Seed 2
0.0000
1.0000
0.1531
0.9854
Seed
0.0000
1.0000
0.1229
0.9855
Seed 4
0.0000
1.0000
0.1108
0.9865
Seed
0.0000
1.0000
0.1443
0.9871
MERGEMANY
0.0141
0.9952
0.0727
0.9831
Table 3: Merging multiple models decreases test loss by 43%. We train five separate MLPs on MNIST. Using Algorithm B we merge all these models together simultaneously. This produces model that appears to have better out-of-distribution performance than any of the input models, with superior test loss performance. We are excited by potential applications of this methodology in federated learning and ensembling, esp. along the lines "model soups" (Wortsman et al. 2022a).
tion than any of the input models. For example, we present the results of this algorithm on MLPs trained on MNIST in Table A.10
In addition, found that merging multiple models helps calibrate the resulting model predic- tions. We present this effect in Figure
A.11 FAILED IDEA: A METHOD FOR STREPEST DESCENT
Imagine standing in weight space O and trying to decide in which immediate direction to move
in order to approach a θ eg-qquivllent point. There are many, many possible permutations of θ
call them 7 (O), π (2) (2) (Op), ... to aim for the distance. Assuming that the loss landscape in fact convex modulo these permutation symmetries, a natural choice would be pick the n. (+) (O)) that corresponds to the direction of steepest descent starting from O since we expect (6) (QB) to lie in the same basin as A. In other words,
dL(AA + ((T(OB) aa))
min
= =min VL(OA) (T(OB) - -04)
(11)
y
A=
VL(O,) min (x(A)))(((1)))
(12)
Now, we are tenuously a favorable situation: VL(OA) is straightforward to compute, and picking the best reduces matching problem. In particular it is a SOBLAP matching problem of the same form as in Section 3.2 In addition, there is a fast, exact solution for the single intermediate
layer case (L 2).
In practice, we found that this method can certainly find directions of steepest descent, but that they accompanied by high barriers in between the initial dip and (Ox).
27

---

Published as a conference paper at ICLR 2023
MergeMany Calibration
Train
Test
1.00
1.00
......
Perfect calibration
Individual models
True probability
Naïve merging
True probability
0.75
Model ensemble
MergeMany
0.50
0.50
0.25
0.25
0.00
0.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Predicted probability
Predicted probability
Figure 13: Merging multiple models results in superior calibration. Here we show the results of running Algorithm 3 32 MLP models trained on MNIST, with each model given access to ran- dom 50% of the trainnn dataset. The resulting merged model demonstrates substantively improved calibration of probability estimates on both the training and test datasets. MergeMany calibration results are competitive with model ensembling, despite requiring 32x less memory and compute.
A.12 PROOF OF LEMMA
To lighten notation (.,)
P in this section.
Lemma. Given A, B E Rdxd,
min
(PAQ B)
P.Q perm, matrices
is strongly NP-hard and has no PTAS.
Proof. We proceed by reduction from the quadratic assignment problem (QAP) (Koopmans & Beck-
mann
1957
Cela 2013). Consider QAP,
min pern.
(PCP D)
for C, D Rdxd
Now, pick A C + XI, B D -I The we have,
XI.
min +P(C+AJ) D XI) (PCQ + APQ D XI) P.Q.
(13)
(PCQ D) XPCQ .T) + xPQ D) (PQ
II)
(14)
(PCQ D) - XP + D X2 tr
(15)
For sufficiently large X. the TPQ) term will dominate.
Letting
a
max (maxi, (Ctgle maxi,j [Dis)), we can bound the other terms,
dda² < (PCQT, DD < da²
(16)
-daa < AMPTQ,C < Ada
(17)
-Ada S XPQT, ,D) < Ada
(18)
Now there are classes of solutions: those where P and those where P * Q. We seek to make the best (lowest) possible P * Q solution to have worse (higher) objective value than the worst (highest) P p= Q solution. When P Q. the highest possible objective value is
daa 2 Xda Ada x²d
28

---

Published as a conference paper at ICLR 2023
and similarly, the lowest possible objective value when p is
adaa² Ada Ada A2d + X2
where the final term is due to the fact that at least one entry PQ must be O. With some algebra, it be that À 5da is sufficient guarantee that all p Q solutions superior to all
P + Q solutions.
Now when P Q. all frivolous terms reduce constants and left with the QAP objective:
min (PCP DD MPTP.OPP + X(PP D) (rr((PP))
--xtr(C) + AtrDD) A + min (PCP), D)
P
completing the reduction. QAP is known to be strongly NP-hard (Koopmans & Beckmann 1957
Sahni & Gonzalez 1976) and MaxQAP is known to not admit any PTAS (Makarychev et al. 2014). thus completing the proof.
A.13 PROOF LEMMA
Lemma. Algorithm 7 Terminates.
Proof. We proceed by contradiction.
Consider graph with each possible permutation edges n; - Ty if πj can be reached from т; with a single P. update, as in Algorithm 1 (Ignore those updates that result in change to Pa in order avoid π, æ, cycles.) Let p(T)) vec(@A)
[PI. PL-13 vertex and directed
vec(T(@gg) denote the utility of a particular л. Note that ग. + ny implies p(T.)) < p(T,). There exist finitely many possible permutations n., meaning that failure terminate must involve cycle in the graph Ti - ..... + T1, However forms total order on the vertices and therefore we have a contradiction.
29


Reviews:
[{"review_id": "YqXv-fj-P0-", "reviewer": "Reviewer_Nc4c", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates the linear mode connectivity of neural networks trained with stochastic gradient descent, i.e., the possibility of smoothly interpolating between two (equivalence classes of) networks trained with different initialization parameters (i.e., one network is permuted to lie in the same loss basin as the other). To that end, the paper proposes three novel algorithms that compute a network permutation that matches another network's behavior (e.g., activations or weights). The approaches are based on regression between activations, approximate weight matching, and straight-through estimation. Finally, the paper conducts an empirical investigation of various architectures and datasets to demonstrate that the proposed methods successfully interpolate between two networks without increasing the loss.", "summary_of_the_review": "Given the significance of the results, the potential impact on the community, and the quality of the paper, I strongly recommend acceptance of the paper.", "strength_and_weaknesses": "**Strengths**\n\nThe paper investigates an essential question in optimizing neural networks: Why do many local minima exist in the loss landscape with similar performance? The paper provides strong evidence for the conjecture that these minima all correspond to the same network modulo permutation symmetries, which provides a significant boost to the community's understanding of gradient descent landscapes. To that end, the paper introduces three novel and computationally efficient permutation selection methods that permute a model's activations or weights to match a target network's behavior. The methods are grounded in concepts from combinatorial optimization and are equipped with convergence guarantees. The paper opens up a variety of exciting research directions:\n* Can the methods be extended for models with slightly different architectures?\n* Does the method also work for models trained on fundamentally different datasets?\n* Can we leverage these methods to combine different networks specializing in specific tasks to create a network capable of performing all tasks simultaneously?\n* Can we leverage these methods to interpolate between two models in a subtractive manner, e.g., subtract a biased model from an accurate model without loss in accuracy?\n\n**Weaknesses**\n\nThe paper claims that `neural network loss landscapes contain (nearly) a single basin after accounting for all possible permutation symmetries of hidden units`. However, the paper only demonstrates that the proposed permutation selection methods induce linear mode connectivity for two given networks (as far as I can tell from the experiment description, the results have not been aggregated over multiple independent runs). As a result, the paper can only reasonably claim that `for some networks, the permutation symmetries allows projecting them into the same loss basin`. Thus, the claim either has to be reformulated, or the experiments have to be repeated for many network initialization parameters to provide evidence for the claim in its current form.\n\nMoreover, some experimental details could be more precise (see clarity below). Finally, the paper posits a hypothesis on the width of models for ImageNet in section 5.1, which is relatively straightforward to test, but does not conduct a corresponding investigation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "REzhmQiX38L", "reviewer": "Reviewer_fSNF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides further evidence for the conjecture from Entezari et. al. 2021 that solutions found by SGD are linearly mode connected modulo permutations. They provide three methods to find these permutations, one of which (weight matching) is very fast to compute. Using these methods they show that high-width neural networks achieve low loss barriers for tasks like MNIST and CIFAR-10.", "summary_of_the_review": "I think this paper studies an important conjecture and clearly provides evidence in support of it. I recommend this paper for acceptance. ", "strength_and_weaknesses": "Strengths:  \n- This paper studies an important open conjecture from Entezari et. al. and provides substantial evidence for it, while stating caveats clearly (requirement of high width)     \n- The proposed algorithms to find the permutations are interesting. I am surprised that the weight matching algorithm works at all, since it can speed up the compute required to solve the permutation problem by a large amount.    \n- The paper is well-written   \n\n\nWeaknesses:    \n- I find the claims regarding the practical utility of merging models trained in a distributed way slightly exaggerated. You don’t see the same improvements in the combined model’s accuracy (Figure 9), and the combined model does much worse than ensembling. Moreover, even ensembling would be a weak baseline if you are trying to combine many models that have been trained on smaller splits of the data. Thus, it is hard for me to see how this method could be helpful in federated learning or distributed training. Nevertheless, I don’t think this particular claim is central to the paper and the results are interesting anyway.  \n\nNeutral questions  \n- Do the authors have any intuitions for why Figure 3b shows so many bumps? Perhaps the training of one of the models is slower? Maybe the right comparison would be interpolating models of approximately equal loss even if they are slightly shifted epochs.    \n- What are the two different lines in each panel of Figure 2?   \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BeHmIZLFsDF", "reviewer": "Reviewer_qnqB", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes 3 methods for permuting the weights of a neural network $A$ to closely match a second neural network $B$. With this procedure, they show that often they achieve linear mode connectivity between the permuted model $A$ and model $B$. They further show that it is possible to merge independently trained models $A, B$ by averaging the weights of the permuted model $A$ and model $B$.", "summary_of_the_review": "Overall, really exciting paper. I pointed out some relatively minor weaknesses, so I currently vote for 8.\n\n## References\n\n[1] [_Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs_](https://arxiv.org/abs/1802.10026);\nTimur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, Andrew Gordon Wilson\n\n[2] [_Essentially No Barriers in Neural Network Energy Landscape_](https://arxiv.org/abs/1803.00885);\nFelix Draxler, Kambis Veschgini, Manfred Salmhofer, Fred A. Hamprecht\n\n[3] [_Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets_](https://arxiv.org/abs/1906.06247);\nRohith Kuditipudi, Xiang Wang, Holden Lee, Yi Zhang, Zhiyuan Li, Wei Hu, Sanjeev Arora, Rong Ge\n\n[4] [_Averaging Weights Leads to Wider Optima and Better Generalization_](https://arxiv.org/abs/1803.05407);\nPavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, Andrew Gordon Wilson\n\n[5] [_Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling_](https://arxiv.org/abs/2102.13042);\nGregory W. Benton, Wesley J. Maddox, Sanae Lotfi, Andrew Gordon Wilson\n\n[6] [_Optimizing Mode Connectivity via Neuron Alignment_](https://arxiv.org/abs/2009.02439);\nN. Joseph Tatro, Pin-Yu Chen, Payel Das, Igor Melnyk, Prasanna Sattigeri, Rongjie Lai\n\n[7] [_Model Fusion via Optimal Transport_](https://arxiv.org/abs/1910.05653);\nSidak Pal Singh, Martin Jaggi\n\n[8] [_Linear Connectivity Reveals Generalization Strategies_](https://arxiv.org/abs/2205.12411);\nJeevesh Juneja, Rachit Bansal, Kyunghyun Cho, João Sedoc, Naomi Saphra\n\n[9] [_Sharp Minima Can Generalize For Deep Nets_](https://arxiv.org/abs/1703.04933);\nLaurent Dinh, Razvan Pascanu, Samy Bengio, Yoshua Bengio\n\n[10] [_On Calibration of Modern Neural Networks_](https://arxiv.org/abs/1706.04599);\nChuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger", "strength_and_weaknesses": "## Strengths\n\nThis is a genuinely exciting paper. I believe it will inspire further research into understanding the loss landscapes as well as merging models and related areas. I think the paper can potentially be quite impactful. \n\nAs I will argue below, I think many of the pieces of this work have been explored before. To me, the main strength of this paper is that it provides a new perspective, or way of thinking about the loss landscapes, even if somewhat obvious in retrospect.\n\n## Weaknesses\n\nI like this paper, so the weaknesses that I point out are somewhat nit-picky.\n\n**W1**: Many of the key pieces of this work have been considered before separately. I think some additional discussion of related work is needed.\n\n**W2**: It appears that the linear mode connectivity results may be somewhat brittle.\n\n**W3**: Some of the interpretations about the SGD inductive biases are not very well supported by the experiments in my opinion.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%