Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivity
Authors: Zhanpeng Zhou
Yongyi Yang
Xiaojiang Yang
Junchi Yan
Wei Hu
Venue: NeurIPS 2023 poster (2023)
Keywords: Linear Mode Connectivity
Permutation Invariance
Optimization Landscape
Science of Deep Learning
TL;DR: We discover a new notion of linear connectivity, Layerwise Linear Feature Connectivity (LLFC), which significantly extends Linear Mode Connectivity (LMC) to features in all layers.

Abstract: Recent work has revealed many intriguing empirical phenomena in neural network training, despite the poorly understood and highly complex loss landscapes and training dynamics. One of these phenomena, Linear Mode Connectivity (LMC), has gained considerable attention due to the intriguing observation that different solutions can be connected by a linear path in the parameter space while maintaining near-constant training and test losses. In this work, we introduce a stronger notion of linear connectivity, Layerwise Linear Feature Connectivity (LLFC), which says that the feature maps of every layer in different trained networks are also linearly connected. We provide comprehensive empirical evidence for LLFC across a wide range of settings, demonstrating that whenever two trained networks satisfy LMC (via either spawning or permutation methods), they also satisfy LLFC in nearly all the layers. Furthermore, we delve deeper into the underlying factors contributing to LLFC, which reveal new insights into the permutation approaches. The study of LLFC transcends and advances our understanding of LMC by adopting a feature-learning perspective.

Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivity
Zhanpeng Zhou', Yongyi Yang", Xiaojiang Yang', Junchi Yan'; Van';Wel Wei Hu
Dept. of Computer Science and Engineering & MoE Key Lab of AI, Shanghai Jiao Tong University Dept. of Electrical Engineering & Computer Science. University of Michigan (zzp1012, y yanjchhi jujjtu fteeuuuu..u... {yongyi, vvh)@umich. edu
Abstract
Recent work has revealed many intriguing empirical phenomena in neural network training, despite the poorly understood and highly complex loss landscapes and training dynamics. One of these phenomena, Linear Mode Connectivity (LMC), has gained considerable attention due to the intriguing observation that different solutions can be connected by linear path in the parameter space while maintaining near-constant training and test losses. In this work, we introduce a stronger notion of linear connectivity, Layerwise Linear Feature Connectivity (LLFC), which says that the feature maps of every layer in different trained networks are also linearly connected. We provide comprehensive empirical evidence for LLFC across wide range of settings, demonstrating that whenever two trained networks satisfy LMC (via either spawning permutation methods), they also satisfy LLFC in nearly all the layers. Furthermore, we delve deeper into the underlying factors contributing approaches. The study of LLFC transcends and advances our understanding of LMC by adopting a feature-learning perspective. We released our source code bttps://gitthu wwt.coo////.a/. cotp:b/1////mmp12/m1..
LLFC, which reveal new insights into the permutation
Introduction
Despite the successes of modern deep neural networks, theoretical understanding of them still lags behind. Efforts understand the mechanisms behind deep learning have led significant interest in exploring the loss landscapes and training dynamics. While the loss functions used in deep learning are often regarded as complex black-box functions in high dimensions, it is believed that these functions, particularly the parts encountered in practical training trajectories, contain intricate benign structures that play role in facilitating the effectiveness of gradient- based training 33 like in many other scientific disciplines, crucial step toward formulating comprehensive theory of deep learning lies meticulous empirical investigations of the learning pipeline, intending to uncover quantitative and reproducible nontrivial phenomena that shed light the underlying mechanisms.
Just
Different optima found by independent runs of gradient-based optimization are connected by a simple path in the parameter space, on which the loss or accuracy nearly constant. This is surprising as different optima of a non-convex function can very well reside in different and isolated "valleys" and yet this does not happen for optima found in practice. More recently, an even stronger form of mode
One intriguing phenomenon discovered in recent work is Mode Connectivity [0 5
Conresponniig author. SJTU authors are partly supported by NSFC(62222607, 61972250, U19B2035) and Shanghai Municipal Science and Technology Major Project 22000022Z00000) Wei Hu acknowledges support from the Google Research Scholar Program.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Feature maps of A
Feature maps of interpolated model 0
Feature maps of Bg
f(o) ixi
Feature maps model proportional to linear interpolation of fealure maps model 8x and fg. where aa + a)
α a
+(1-a) a1
Figure 1:
Illustration of Layerwise Linear Feature Connectivity (LLFC). Each tensor comprises three feature maps when provided with three input images.
connectivity called Linear Mode Connectivity (LMC) discovered [24 9 It depicts that different optima can be connected by a linear path constant loss/accuracy. Although LMC typically does happen for two independently trained networks, it has been consistently observed the following scenarios:
Spawning 9.77 7 A network is randomly initialized, trained for small number of epochs (e.g. 5 epochs for both ResNet-20 and VGG-16 on the CIFAR-10 dataset), and then spawned into two copies which continue to be independently trained using different SGD randomnesses (i.e., for mini-batch order and data augmentation).
Permutation 6 I:
Two networks are independently trained, and the neurons of one model permuted match the neurons of the other model while maintaining functionally equivalent
network.
The study of LMC is highly motivated due to its ability to unveil nontrivial structural properties of loss landscapes and training dynamics. Furthermore, LMC has significant relevance to various practical applications, such pruning and weight-averaging methods.
On the other hand, the success of deep neural networks is related to their ability to learn useful features, or representations, of the data [27]. and recent work has highlighted the importance of analyzing not only the final outputs of a network but also intermediate features (0]. However, this crucial perspective is absent in the existing literature LMC and weight averaging. These studies typically focus on interpolating the weights different models and examining the final loss and accuracy, without delving into the internal layers of the network.
In this work, we take reature-learning perspective on LMC and pose the question: what happens to the internal features when we linearly interpolate the weights of two trained networks? Our main discovery, referred as Layerwise Linear Feature Connectivity (LLFC), that the features in almost all the layers also satisfy a strong form of linear connectivity: the feature map in the weight-interpolated network is approximately the same as the linear interpolation the feature maps in the two original networks. More precisely, let 0 and 0g be the weights of two trained networks, and let (0) be the feature map in the é-th layer of the network with weights 0. Then we say that A and 0g satisfy LLFC
(a0 + a)) (0s) + (1))))(00))))
Va E [0, 1J.
(I)
See Figure for illustration. While LLFC certainly cannot hold for two arbitrary O and O find that it is satisfied whenever A and 0g satisfy LMC. We confirm this across wide range of settings, as well as for both spawning and permutation methods that give rise to LMC.
LLFC is much finer- iner-grained characterization of linearity than LMC. While LMC only concerns loss or accuracy, which is single scalar value, LLFC (I) establishes relation for all intermediate feature maps, which are high-dimensional objects. Furthermore, the output layer implies LMC when the two networks have small emrors (see Lemma I hence, LLFC can be viewed as strictly stronger property than LMC. The consistent co-occurrence of L.LFC and LMC suggests that studying LLFC may play crucial role in enhancing our understanding of LMC.
not difficult to see that LLFC applied to

---

Subsequently, we delve deeper into the underlying factors contributing to LLFC. We identify two critical conditions, weak additivity for ReLU function and commutativvty property between two trained networks. We prove that these two conditions collectively imply LLFC in ReI.U networks, and provide empirical verification of these conditions. Furthermore, our investigation yields novel insights into the permutation approaches: we interpret both the activation matching and weight matching objectives in Git Re-Basin [] as ways to ensure the satisfaction of commutativity property.
In summary, our work unveils a richer set of phenomena that go significantly beyond the scope of LMC, and our further investigation provides valuable insights into the underlying mechanism behind LMC. Our results demonstrate the value of opening the black box of neural networks and taking feature-centric viewpoint in studying questions related loss landscapes and training dynamics.
2 Related Work
(Linear) Mode Connectivity. Freeman and Bruna [0], Draxler et al. [5]. Garipov et al. 11 observed Mode Connectivity, i.e., different optima/modes of the loss function can be connected through non-linear path with nearly constant loss. Nagarajan and Kolter [24] first observed Linear Mode Connectivity (LMC), i.e., the near-constant-loss connecting path can be linear, on models trained on MNIST starting from the same random initialization. Later, Frankle et al. [9] first formally defined and thoroughly investigate the LMC problem. Frankle al. [9] observed LMC on harder datasets, for networks that are jointly trained for short amount of time before going through independent training (we refer to this as the spawning method). Frankle et al. [9] also demonstrated a connection between LMC and the Lottery Ticket Hypothesis [8] Fort et al. 7 used the spawning method to explore the connection between LMC and the Neural Tangent Kernel dynamics. Lubana et al. (23 studied the mechanisms of DNNs from mode connectivity and verified the mechanistically dissimilar models cannot be linearly connected. Yunis et al. (38 showed that LMC also extends beyond two optima and identified high-dimensional convex hull of low loss between multiple optima. On the 26
theory side, several papers [T 21 32
25 16 were able to prove non-linear mode connectivity
under various settings, but there has not been a theoretical explanation of LMC to our knowledge.
Invariance. Neural network architectures contain permutation symmetries [3] one can permute the weights in different layers while not changing the function computed by the network. Ashmore and Gashler 2 utilized the permutation invariance of DNNs to align the topological structure of two neural networks. Tatro et al. [3 used permutation invariance to align the neurons of two neural networks, resulting improved non-linear mode connectivity. In the context of LMC. Entezari et al. [6. Ainsworth et al. [I] showed that even independently trained networks can be linearly connected when permutation invariance is taken into account. particular, Ainsworth et al. 11 approached the neuron alignment problem by formulating it as bipartite graph matching and proposed two matching methods: activation matching and weight matching. Notably, Ainsworth et al. []] achieved the LMC between independently trained ResNet models on the CIFAR-10 dataset using weight matching.
Permutation
Model Averaging Methods. LMC also has direct implications for model averaging methods, which are further related federated learning and ensemble methods. Wang al. [33] introduced a novel federated learning algorithm that incorporates unit permutation before model averaging. Singh and Jaggi (00, Liu et al. (22) approached the neuron alignment problem in model averaging by formulating it as an optimal transport and graph matching problem, respectively. Wortsman et al. 335 averaged the weights of multiple fine-tuned models trained with different hyper-parameters and obtained improved performance.
3 Background and Preliminaries
Notation and Setup. Denote [k] = (1.2...., k]. We consider a classification dataset D {( y:) ) where a; do represents the input and ys [c] represents the label of the i-th data point. Here, n is the dataset size, dg is the input dimension and e is the number of classes. Moreover, use X E dodxx to stack all the input data into matrix.
We consider an L-layer neural network of the form f(0; x)) where 0 represents the model parameters, is the input, and [(0; x) E RR.. Let the ε-th layer feature (post-activation) of the network be f(() (0: x) E di, where de is the dimension of the 6-th layer (0 l L) and di c. Note that
3

---

(0:))) = a and f (0;=) = z)) For an input data matrix X, we also use f(0; E R and f( (0: E dd to denote the collection of the network outputs and features on all the datapoints, respectively. When X is clear from the context, we simply write (0) (0; X) and f()) = f(:: X). Unless otherwise specified, in this paper we consider models trained on training set, and then all the investigations are evaluated on a test set.
We use ErrP (0) to denote the classification error of the network f(0:)) on the dataset DD
Linear Mode Connectivity (LMC).. We recall the notion of LMC. in Definition [
Definition (Linear Mode Connectivity). Given a test dataset 'D and two mode
3 0 and On such
that (0TT ss Erro(On) n), we say 0 A and O n are linearly connected if they satisfy
Errp (a0, A + (1 a)0B) ))== Re EITD(OA):
Va E [0,
(2)
As Definition 1 shows, A and f satisfy LMC if the error metric on the linear path connecting their weights is nearly constant. There two known methods to obtain linearly connected modes, the spawning method [ 7
and the permutation method [6
Spawning Method. We start from random initialization (()) and train the model for k steps to obtain (k)) Then we create two copies of (*) and continue training the two models separately using independent SGD randomnesses (mini-batch order and data augmentations) until convergence. By selecting proper value of k (usually a small fraction of the total training steps), we can obtain two linearly connected modes.
Permutation Method. Due to permutation symmetry, it is possible to permute the weights in a neural network appropriately while not changing the function being computed. Given two modes O and f which are independently trained (and not linearly connected), the permutation method aims find a permutation such that the permuted mode 0' x00 8) is functionally equivalent to 0g and that o'z and O are linearly connected. In other words, even if two modes are not linearly connected in the parameter space, they might still be linearly connected if permutation invariance is taken into account.
Among existing permutation methods, Git Re-Basin [ is representative one, which successfully achieved linear connectivity between two independently trained ResNet models on CIFAR-10 10. Specifically, permutation that maintains functionally equivalent network can be formulated by set of per-layer permutations { /-=1 where p(() [Rd x is permutation matrix. Ainsworth
et al. (1) proposed two distinct matching objectives for aligning the neurons of independently trained models via permutation: weight matching and activation matching:
Weight matching:
min ||0 - T (0
(3)
Activation matching:
min
H
P(O H (
(4)
Here, A = (0 X) is the f-th layer feature matrix, and H is defined similarly.
B
both the spawning method and the permutation method to obtain linearly connected modes. Following Frankle et al. (.. Ainsworth et al. [. we perform experiments commonly used image classification datasets MNIST [8], CIFAR- 10 [1], and Siny-ImageNe([T 17], and with the standard network architectures 200/500-0050 [2]. VGG-16 29), and MLP. We follow the same training procedures and parerepprreeters in Frankle et al. [9. Ainsworth et al. M. Due to space limit, we defer some of the experimental results to the appendix. Notice that [1] increased the width of ResNet by 32 times in order to achieve zero barrier, and we also followed this setting in the experiments of the permutation method. The detailed settings and typer-parameten are also described in Appendix B.
Main Experimental Setup. In this paper,
4 Layerwise Linear Feature Connectivity (LLFC)
In this section, we formally describe Layerwise Linear Feature Connectivity (LLFC) and provide empirical evidence of its consistent co-occurrence with LMC. We also show that LLFC applied to the last layer directly implies LMC.
Following the terminology in literature, mode refers an optimal solution obtained at the end of training.

---

Enst cosine(::] Ey[1 cosine(k))) Eoft cosine (R)]]
MLP on M NNST
ResNe120 on CTFA R10
1.0
0.5
PC
PC
PC3
FC44
Block 1-2
Block 1-3
Block 2-2
Block 2-3
Block 3-2
Block 3-3
PC
ResNet50 on Tiny ImagaNet
1.4
Block 2-4
Blook 3-1
Block 3-2
Block 3-3
Block Block Block
Block
Block
Block
PC
VGG16 on CIBAR10
1.0
0.0
cosime
cosineuzs
cosinty sa
cosineeus
Figure 2: Comparison of [1[[ - cosine, (ax;) and Ep 1 cosine A. (a:)]. The spawning method
used to obtain two linearly connected modes A and 0 Results are presented for different layers of various model architectures on different datasets, with a E {0.25, 0.5, 0.75). Standard deviations across the dataset are reported by error bars. More results are in Appendix B2
coxina(C)) cx0ine(x))
(Aclivation Matching)
Soll. cosine((x)) Egl1 cosineee))
on (weight matching)
0.5
FC
FC
FC
FC
FC
FC
MLP on CIFAR-10 (A (Activation Matching)
MLP CIFAR-10
(weight matching)
0.0
FC1
FC2
0.0
PC2
FC3
FC
Ap13 cosino(x;]]
ReaNet20 (32 x) CIRAR-10 (Weight Matching)
1.0
0.0
Rlock
Rlock
Block
cosine
costneuzs
cosineuso
cosineurs
Figure 3: Comparison of Ep[1- [tp[l-cosine,
cosine, and Ep[1- cosine (x,)) The permutation method
used to obtain linearly connected modes A and 0g. Results are presented for different layers of various model architectures on different datasets, with a E {0.25, 0.5, 0.75). Standard deviations across the dataset reported by error bars. More results are in Appendix B.2
Definition 2 (Layerwise Linear Feature Connectivity). Given dataset D and two modes O OB of an L-layer neural network f. the modes O and 0g are said to be layerwise linearly feature connected if they satisfy
Vf E [Z], Va E 10, Be O.s.t (a00 (1- - o (0) (5)
LLFC states that the per-layer feature of the interpolated model 0,, a0 (1 0 has the same direction as the linear interpolation the features of A and fg. This means that the feature map f(() behaves similarly to linear map (up to a scaling factor) on the line segment between A and 0 even though it is nonlinear map globally.
LLFC Co-occurs with LMC. We now verify that LLFC consistently co-occurs with LMC across different architectures and datasets. We use the spawning method and the permutation method described in Section 3 to obtain linearly connected modes 0 a and Q.. On each data point æ; the test set D. measure the cosine similarity between the feature of the interpolated model 0,, and
5

---

Eng cosine(k;]]
ResNer20 on CTFARIO (Sawning)
Res (32x) on CIARRO (Wäght Metching)
1.4
0.5
23
3-1
Block 32
Block
2l3
Block3-1
Block32
Block33
PC
cosine LMC
cosinesae
Figure Comparison of [1 cosiner.M and [Ep[1 cosinemot LMC (x;)) Both the spawning and permutation methods are used to obtain two linearly connected modes. Standard deviations across the dataset are reported by error bars.
linear interpolations of the features of A and A in each layer f, as expressed as cosine, (an)) (os[f(()(0) A + (1 a) æ;), (0 æ.) ) + (1 f() æ;)]. We compare this to the baseline cosine similarity between the features of O and On in the corresponding layer, namely cosineA.B(x)) = cos(f()) (;); f()
)). The results for the spawning method and the permutation method are presented in Figures 2 and respectively. They show that the values of Ep[1 - Cosineo (x:)) are close to 0 across different layers, architectures, datasets, and different values of a, which verifies the LLFC property. The presence of small error bars indicates consistent behavior for each data point. Moreover, the values of Ep [1 cosine (2x)) are not close to O, which rules out the trivial case that x0) and f( BB) are already perfectly aligned.
To further verify, we also compare the values Ep Ep[1-cosine, cosine., of two linearly connected modes with those of two modes that are independently trained (not satisfying LMC). We measure cosineo.s of the features of two modes that are linearly connected and two modes that are independently trained, denoted as cosineLMC and cosine LMC correspondingly. In Figure 44] the values of
Ep [1 c are negligible compared Ep [1 cosinc LMC (ar;). The experimental
results align with Figures 2 and 3 and thus we firmly verify the claim that LLFC co-occurs with LMC.
LLFC Implies LMC. Intuitively, LLFC is much stronger characterization than LMC since establishes linearity property in the high-dimensional feature map in every layer, rather than just for the final error. Lemma[] below formally establishes that LMC a consequence of LLFC by applying LLFC on the output layer.
Lemma (Proof in Appendix A.IIp Suppose two modes O, Og satisfy LLFC on dalaset 'D and max (ETTD(( Errp ( B)} I then we have
Va E [0, 1], Errp(ar
a)0 S 2c.
(6)
In summary, we see that the LMC property, which was used study the loss landscapes in the entire parameter space, extends to the internal features in almost all the layers. LLFC offers much richer structural properties than LMC. In Section we will dig deeper into the contributing factors LLFC and leverage the insights to gain new understanding of the spawning and the permutation methods.
5 Why Does LLFC Emerge?
We have that LLFC is a prevalent phenomenon that co-occurs with LMC, and establishes a broader notion of linear connectivity than LMC. In this section, we investigate the root cause of LLFC, and identify two key conditions, weak additivity for ReLU activations and commutativiry. We verify these conditions empirically and prove that they collectively imply LLFC. From there, we provide an explanation for the effectiveness of the permutation method, offering new insights into LMC.
5.1 Underlying Factors LLFC
For convenience, consider multi-layer perceptron (MLP) in Section though the results can be easily adapted to any reed-forward structure, e.g., a convolutional neural network n (CNN). For
Conv layer. forward propagation will be de- noted as W H similar to a linear layer. Typically. the weight W for a Conv layer has shape (# of output channels, # of input channels, height, width) and we reshape W to matrix with dimensions of output channels, # of input channels x height x width).
*We also conduct experiments on CNNs.

---

MI.P Pn MNIST
VGGI6 on CIBARIO
Telepp DICL
Hodeps Birt,
X
RC1
RC2
FC3
Can +1
Cocy 4-2
Oun/
OZe S-1
Coty 52
Ouwy 5-3
Dist, ah.)
Figure 5: Comparison between the distribution of the normalized distance Dist, (hi,A, ha,n) and Dist, (hi.c, D. Here, h and ha are features of two linearly connected modes, i.e., O and OB (founded by the spawning method). h and h. D comes from modes that are independently trained. Results are presented for different layers MLP on MNIST and VGG-16 on CIFAR-10.
MNIST
ResNet20 (32x) CTFARIO
DotreeDier
Distnoo Din
Doleromee
Dotermois
FC
FC2
FC3
Block H-
Block 2-1
Block
Bloc3-1
Block-22 3-2
Dista B
Dist, benn
Figure Comparison among the distribution the normalized distance Dist, (ha,o. Dea)) Dist, (hi.,, hug. and Dist, (hr.c, h.,p). Here, hic and hD are features of two modes that independently trained, i.e., Oc and 0p. hu..D.ct and h., D.gg comes from D.... (permuted Op using the activation matching) and D (permuted pD using the weight matching), correspondingly, For ResNet-20, the values of Dist, are calculated in the first Conv layer of each block. Results are presented for different layers of MLP on MNIST and ResNet-20 on CIFAR-10.
an L-layer MLP f with ReLU activation, the weight matrix in the é-th linear layer is denoted ( Rdexde-1, and (0) E Id is the bias in that layer. For given input data matrix X [gdexm, denote the feature (post-activation) in the f-th layer as H f(0 X) gdxx"
and correspondingly pre-activation f (C) [Rddxnn The forward propagation in the b-th layer is:
H = (H
= ()) A ( (t-1) 1.
Here, a denotes the ReLU activation function, and 1d, E Id denotes the all-one vector. Additionally, (8)
we use (4) to denote the i-th row of H and h to denote the i-th row of (() which correspond
the post- and pre-activations of the i-th input at layer respectively.
Condition I: Weak Additivity for ReLU Activations
F]
Definition 3 (Weak Additivity for ReLU Activations). Given dataset DD two modes O and Og
are said to satisfy weak additivity for ReLU activations if
Vg E [L], Vo E [0, 1]. (a (1 (6)
No (1 (0)
Definition requires the ReLU activation function to behave like linear function for the pre- activations in each layer of the two networks. Although this cannot be true in general since ReLU is nonlinear function, verify it empirically for modes that satisfy LMC and LLFC.
We conduct experiments on various datasets and architectures to validate the weak additivity for ReLU activations. Specifically, given two modes 0 and 0 and a data point æ; in the test set D. we compute the normalized distances between the left-hand side and the right-hand side of Equation 7 for each layer L, varying the values of a. We denote the maximum distance across the range of
as Dist, (h.,A. ha = max, [o. dist(a (añ4.a + (1 - h),,,) (hi,A) + (1 - a)o (h.,)).
where dist(x; 3)= := ||z y /[|=][ |lyl)). .. To validate the weak additivity condition, we compare
the values of Dist,, of two modes that are linearly connected with those of two modes that independently trained.
'Weak additivity has no relation to stable neurons 28 Stable neuron is defined as one whose output is the constant value zero or the pre- activation output on all inputs. which is property concerning single network. On the other hand, weak additivity concerns relation between networks.
7

---

MI. Pon MINIST
RosNe on CIFARIO
Desano
RC22
RC4
Block 1-2
Black 1-3
Blodk
Block
Block
Flock
MNSST (WediMarrinng Matching)
ResNNe20 (32 x)
CIENROO (Wesl Matching)
D 20000
Deteeos
FC2
FC4
Blodk 3-2
Blode 1-3
Blodk2222
Blodk233
Block322
Block 3-
Distoor
Disky
Disty
Figure 7: Comparison of Dist.om. Disty, and Disty In the first row, the spawning method is used to acquire modes that satisfy LLFC, whereas the permutation method used for the second row. For ResNet-20, Distcom. Distw are calculated in the first Conv layer of each block. The results are presented for different layers MLP on the MNIST and ResNet-20 on the CIFAR-10. More results are in Appendix B.3
Both spawning and permutation methods are shown to demonstrate the weak additivity condition. For spawning method, we first obtain two linearly connected modes, i.e, A and θ and then two independently trained modes, i.c, Oc and On (not satisfying LMC/LLFC). We compare the values of Dist, of A and 0B with those of 0c and 0 p. In Figure 5 we observe that across different datasets and model architectures, different layers, Dist, Dist,(hi,A h. are negligible (and much smaller than the baseline Dist, Dist,(hs,c. )). ). For permutation methods, given two independently trained modes, ie., Oc and Pp. permute the OD such that the permuted л (@p) are linearly connected with Oc- Both activation matching and weight matching are used and the permuted π (θp) are denoted as A D..s. and O D respectively. In Figure 6 the values of Dist, (h,,c, h..D...) and Dist, (h,,c, h, Dwge) are close to zero compared Dist, (hi,c, h., Therefore, verify the weak additivity condition for both spawning and permutation methods.
Condition II: Commutativity.
Definition (Commutativityy. Given dataset D, two modes O and 0B are said satisfy commutativity if
Yl e [L). W H W H w H ( W H
(8)
Commutativity depicts that the next-layer linear transformations applied to the internal features of two neural networks can be interchanged. This property crucial for improving our understanding of LMC and LLFC. In Section 5.2 we will use the commutativity property to provide new insights into the permutation method.
We conduct experiments on various datasets and model architectures to verify the com- mutativity property for modes that satisfy LLFC. Specifically, for given dataset DD and two modes 0Λ and 03 that satisfy LLFC, we compute the normalized distance be- tween the left-hand side and the right-hand side of Equation (8), denoted as Distcom dist (vec(W(() (L-1) w! (((")) vec w H (((-1) W
H "-)) Furthermore, we compare Distcom with the normalized distance between the weight matrices of the current layer f, denoted as Distw, and the normalized distances between the post-activations of the previous layer
e - 1, denoted as Distg. These distances are expressed as Distw = dist (ve((W((() A vec ()()))
and Dist dist (vec(Hr((-)) vec(HH( respectively. Figure shows that for both spawning and permutation methods, Dist.om negligible compared with Distw and Distg, which confirms the commutativity condition. Note that we also rule out the trivial case where either the weight matrices or the post-activations in the two networks are already perfectly aligned, as weights and post-activations often differ significantly. We also add more baseline experiments for comparison (see Appendix B.3 for more results).
Additionally, note that commutativity has similarity to model stitching 19 3 we show that stronger form of model stitching (without an additional trainable layer) works for two networks that satisfy LLFC.
In Appendix B.4

---

Conditions I and II Imply LLFC. Theorem below shows that weak additivity for ReLU activations
(Definition B] and commutativity (Definition imply LLFC.
Theorem 1 (Proof in Appendix A.2). Given dataset DD if two modes 0. and OB satisfy weak
additivity for ReLU activations (Defmittion 3 and commutativity (Definition 4. then
Va E [0, 1], ve E [L], ) (a00 + n)) - a (0x) + +1 a1) (() (On).
Note that the definition of LLFC (Definition 2 allows a scaling factor while Theorem estabbishes a stronger version LLFC where с = 1. We attribute this inconsistency to the accumulation of error in weak additivity and commutativity conditions, since they are only approximated satisfied practice. Yet in most cases, we observe that is close to (see Appendix [3.2 for more results).
5.2 Justification of the Permutation Methods
In this subsection, we provide a justification of the permutation methods in Git Re-Basin [: weight matching and activation matching (4) Recall from Section 3 that given two modes 0 and 0g. the permutation method aims to find a permutation O'B - -= (0 and A are linearly connected, where p(() is a permutation matrix applied to the L-th layer feature. Concretely, with a permutation x, we can formulate W
(P Oy
h-1 such that the permuted
(2) and H (2) of f in each layer & as W (2) p(0) p((-1) and H (4) p(t) H B In Section 5.1 we have identified the commutativity property as a key factor contributing to LLFC. The commutativity property (8) between A and B can be written as
(W
W (6)
(H(-1)
H
-H' (1-1)) 0,
or
( p w 4 p(e-1) (L-1) ((61)) H ((-1)
= 0.
(9)
We note that the weight matching and activation matching objectives can be written as min, E
p
w ( p((-1) and min, Σt11 H p H ( respectively,
which directly correspond to the two factors Equation (9) Therefore, we can interpret Git Re-Basin as a means ensure commutativity. Extended discussion on Git Re-basin Appendix B.5
[ can be found in
6 Conclusion and Discussion
We identified Layerwise Linear Feature Connectivity (LLFC) prevalent phenomenon that co- occurs with Linear Mode Connectivity (LMC). By investigating the underlying contributing factors LLFC, we obtained novel insights into the existing permutation methods that give rise to LMC. The consistent co-occurrence of LMC and LLFC suggests that LLFC may play an important role if we want to understand LMC in full. Since the LLFC phenomenon suggests that averaging weights is roughly equivalent to averaging features, natural future direction is to study feature averaging methods and investigate whether averaging leads better features.
We note that our current experiments mainly focus on image classification tasks, though aligning with existing literature on LMC. We leave the exploration of empirical evidence beyond image classification as future direction. We also note that our Theorem predicts LLFC in an ideal case, while practice, scaling factor e is introduced to Definition to better describe the experimental results. Realistic theorems and definitions (approximated version) are defered to future research.
Finally, leave the question if it is possible to find permutation directly enforcing the commutativ- ity property 19) Minimizing ( ( p(0) w ( p(é-1) (H((11) p((-1) H / I) entails
A
solving Quadratic Assignment Problems (QAPs) (See Appendix A.3 for derivation), which are known NP-hard. Solving QAPs calls for efficient techniques especially seeing the progress in learning to
solve QAPs [37 34|.
'we find that employing scaling factor enables much better description of the practical behavior than other choices (e.g. an additive enror term).
Note that both p(0) and p(L) are identity matrices.

---

References
[1]
Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo Eleventh International Conference on Learning Reprexentations 2023.
permutation symmetriess. nett/form"
URL http:/ttt/sseereeiiiiii/// wtt::oorrur.dddddmm/mttaaaaaaa 21=:00mmmm1mm511
[2] Stephen Ashmore and Michael Gashler. A method for finding similarity between multi-layer layer perceptrons by forward bipartite alignment. In 20/5 International Joint Conference Neural Networks (IJCNN). pages 1-7. IEEE, 2015.
[3] Yamini Bansal, Preetum Nakkiran, and Boaz Barak. Revisiting model stitching to compare neural repre- sentations. In A. Beygelzimer, Dauphin, Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL httpp:://opeeeeveeiee httppp///ppeeerrrieeeeeerrrrrrrp pett:forummii-aa0065555555
net
[4] Yuan Cao, Zixiang Chen, Misha Belkin, and Quanquan Gu. Benign overfitting in two-layer convolutional neural networks. Advances in neural information processing systems, 35:25237-25250, 2022.
[5] Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International conference on machine learning, pages 1309-1318. PMLR, 2018.
[6] Rahim Entezari, Hanic Sodghi, Olga Saukh, and Behnam Neyshabur, The role of permutation invariance in linear mode connectivity of neural networks. In International Conference Learning Representations, 2022. URL URL httpps:/oppeerreeeeeeeee
[7] Stanislav Fort, Gintare Karolina Dziugaite. Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. Advances in Neural Information Processing Systems, 33:5850-5861. 2020.
[8] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. International Conference on Learning Representations, 2019. URL httpss//pp//perreeie
[9] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pages 3259-3269. PMIR, 2020.
[10] C. Daniel Freeman and Joan Bruna. Topology and geometry of half-roctified network optimization. In International Conference on Learning Representations, 2017. URU.nttpe://opprreiieee.ee/ee 1d 1d-BkoFWVcgx
[11] Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dans. Advances neural information processing systems, 31, 2018.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016. doi: 10. 10911099...2206.900
[13] Robert Hecht-Nielsen. On the algebraic structure of feedforward network weight spaces. 1990.
[14] Tjalling Koopmans and Martin Beckmann. Assignment problems and the location of economic activities. Econometrica: journal of the Econometric Suciety, pages 53-76, 1957.
[15] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[16] Rohith Kuditipudi,, Xiang Wang, Holden Lee, Yi Zhang, Zhiyuan Li, Wei Hu, Rong Ge, and Sanjeev Arora. Explaining landscape connectivity of low-cost solutions for multilayer nets. In H. Wallach, H. Larochelle. A. Beygelzimer. F. d'Alché-Bar Buc. E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc.. 2019. URL http:: http://://///cceeeeneeeee Paper. -pdf
neurips.co. caper.
[17] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 23/N, (7))3,2005.
[18] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 2222222272 1998.
10

---

[19] Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equivariance and equivalence. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 991-999, 2015.
[20] Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent learning: Do different neural networks learn the same representations?
Dmitry Storcheus, Afshin Rostamizadeh, and Sanjiv Kumar, editors, Proceedings of the 1st International Workshop on Feature Extruction: Modern Questions and Challenges at NIPS 2015, volume 44 of Proceedings of Machine Learning Research, pages 196-212, Montreal, Canada, 11 Dec 2015. PMLR. URL https://proceeddnggg mlr .aar..pres///e v44/ 1i15convergent. html
[21] Shiyu Liang, Ruoyu Sun, Yixuan Li, and Rayadurgam Srikant. Understanding the loss surface of neural networks for binary classification. In International Conference on Machine Learning, pages 2835-2832 2843. PMLR, 2018.
[22] Chang Liu, Chenfei Lou, Runzhong Wang, Alan Yuhan Xi, Shen, and Junchi Yan. Deep neural network fusion via graph matching with applications to model ensemble and federated learning. In ICML, pages 13857- 13869, 2022. URL https:
mlr press/vv62// 111222 html
[23] Ekdeep Singh I Lubana, Eric J Bigelow, Robert P. Dick, David Krueger, and Hidenori Tanaka. Mcchanistic mode connectivity, 2023. URL htttt::///ppereeiisssssse www.:eeeerorrididiioooaoaoaaeeee
[24] Vaishnavh Nagarajan and Zico Kolter. Uniform convergence may be unable to explain generalization in deep learning. Advances in Neural Information Processing Systems, 32, 2019.
[25] Quynh Nguyen. On connected sublevel sets in deep learning. In International conference on machine learning, pages 4790-4799, PMLR, 2019.
[26] Quynh Nguyen, Mahesh Chandra Mukkamala, and Matthias Hein. On the loss landscape of class of deep neural networks with no bad local valleys. In International Conference Learning Representations, 2019. URL httt//tt/pp/pprerrette
[27] David E Rumelhart, Geolfrey E Hinton, and Ronald Williams. Learning internal representations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
[28] Thiago Serra, Xin Yu, Abhinav Kumar, and Srikumar Ramalingam. Scaling up exact neural network compression by reLU stability. In A. Beygelzimer, Dauphin, Liang, and J. Wortman Vaughan, editors. Advances in Neural Information Processing Systems, 2021. URL httl:tttt////oererr// net/forum? id=tqQ-8MuSqm
[29] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Represen- tations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015. URL
/stp:///rxiv..o org/ abs////////1 11/9//1199... 1556
[30] Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. Advances in Neural Information Processing Systems, 33:22045-22055 2020.
[31] Norman Tatro, Pin-Ye Chen, Payel Das, Igor Melnyk, Prasanna Sattigeri, and Rongjic Lai. Optimizing mode connectivity via neuron alignment. Advances in Neural Information Processing Systems, 33:15300- 15311, 2020.
[32] Luca Venturi, Afonso S. Bandeira, and Joan Bruna. Spurious valleys in hhddhhidddyyeee neural network optimization landscapes. J. Mach. Learn. Res., 00:113311133::: 2019. URL. http://jmmr /rrg//r///////o///////// papers/v20/18-a 674. html
[33] Hongyi Wang, Mikhail Yurochkin, Yuckai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. In International Conference Learning Representations, 2020. URL
https://openrevieu net/foorunidd-Bluueee
[34] Runzhong Wang, Junchi Yan, and Xiaokang Yang. Neural graph matching network: Learning lawler's quadratic assignment problem with extension to hypergraph and calttiple-grahh matching. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 2022.
[35] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: aver- aging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference Machine Learning, pages 23965 2099223998888 PMLR, 2022.
11

---

[36] Zhiwei Xu, Yutong Wang, Spencer Frei, Gal Vardi, and Wei Hu. Benign overfitting and grokking in relu networks for cluster data. arXiv preprint arX1n:2310.0244, 2023.
[37] Tianshu Yu, Junchi Yan, Yilin Wang, Wei Liu, and baoxin Li. Generalizing graph matching heyond guadratic assignment model. In S. Bengio. H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Cesa-Bianchi, and Garnett, editors, Advances Neural Information Processing Systems. volume 31. Curran sociates, Inc., 2018. URL 5te52bbe.c60a1cc..d2Zeeee wtt.:/222555e77755555555225 Papes--pprpppp pdf
neurips. http://pppee.iieeeepeeeeee//eeee
[38] David Yunis, Kumar Kshitij Patel, Pedro Henrique Pamplona Savarese, Gal Vardi, Jonathan Frankle, Matthew Walter, Karen Livescu, and Michael Maire. On convexity and linear mode connectivity in neural networks. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop). 2022. URL httpa://openrevieu wwt.:foorumiSA47000X333PPP
[39] Zhenyu Zhu, Fanghui Liu, Grigorios Chrysos, Francesco Locatello, and Volkan Cevher. Benign overfitting in deep neural networks under lazy training. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 43105-43128. URL WRRLhttps:/// htttp::/ppoceeediiiiii mlr halr.prss/ ppppp//202/ v202/ zhu23h. html
PMLR, 23-29 Jul 2023.
12

---

A Proofs of Theorems and Derivations
A.1 Proof of Lemma
In this section, we prove Lemma I in the main paper. This lemma indicates that we can directly imply Linear Mode Connectivity (LMC, see Definition [ from Layerwise Linear Feature Connectivity (LLFC, see Definition 2 applied to last layer.
Definition (Linear Mode Connectivity). Given a test dataset D and two modes O and O such that ETTD A0 ErTD (0B), we say A and B are linearly connected if they satisfy
Errp (a0 + a))0s) EIID (0A),
Ya E [0.1].
Definition 2 (Layerwise Linear Feature Connectivity). Given dataset 'D and two modes A OB of an L-layer neural network f, the modes 0 A and Og are said to be layerwise linearly feature connected if they satisfy
vl E [L], Vo E [0, 3e O,s.t. cf (a0 + (1 01) af( (0A) + + f(0) (0B)
Lemma 1. Suppose modes 0 On satisfy LLFC dataset D and
max {ErrD (0 Errp (0B)) S
then we have
Va E [0, 1], ErrD (a0 + (1 a) S 2c.
Proof. Note that the classification depends on the relative order the entries the output of the final layer. As a consequence, for each data point in the dataset DD the linear interpolation of the outputs of the models makes the correct classification if both models make the correct classification. Therefore, only if one of the model makes the incorrect classification, the linear interpolation of the outputs of the models would possibly make the incorrect classification, i.e,
ErTp (af(0s) + - a) f(0))) ErTD (0A) ErTD (0B).
Since 0 and O satisfy LLFC, then at last layer we have f (a0 + + a1-a) = af (1-
then have
Errp (a0, + +1 (1 (1-a))pp)) S Erro (0x) + ERrD (0n).
According to the condition that
max (ErrD (0x), Frrp (0n)) S
which indicates
Enp (a0A + +(1 - a.) 0B)) < 2e,
and this finishes the proof.
A.2 Proof of Theorem
In this section, we prove Theorem in the main paper. Theorem indicates that we can derive LLFC from two simple conditions: weak additivity for ReLUU activations (Definition 3 and commutativity (Definition 4. Note that though we consider a multi-layer perceptron (MLP) for convenience, our proof and results can be easily adopted to any feed-forward structure, e.g., a convolutional neural network (CNN).
Definition (Weak Additivity for ReLU Activations). Given dataset D, two modes O and Og are said to satisfy weak additivity for ReLU activations if
vl E (L),
(H
AA = (H + [H (2)
13

---

Definition 4 (Commutativiit), Given dataset D. two modes A and 0g are said to satisfy
commutativity if
e [L]
W 1 H + B
w H H + (
B
Theorem 1. Given a dataset D. if two modes O and 0g satisfy weak additivity for ReLU activations
(Definition 3) and commulativity (Definition 1. then
Ya E [0. VE E [4], f(0) (a0A + (1 (1-100) a) 0B) = a (0x) + (1) f (0n).
Proof. Before delving into the proof, let us denote the forward propagation in each layer & by
gl 0: H() w(0) + I. T
g0 0: (0:H1(-1))
=o (5'0 (0: (0:11(--1)) (1
Given A and 0g that satisfy the commutativity property, then AE E [L] and Va E [0. 1]. we have
w A ((-1))
(0) ((-1) =w
H ( + (
B H (1-1)
+
B
B
A
800 (0::
+ +
0 H
B 1) =0 (0A: + g())
(0g: H (-11)
A
a (1- (500 (OA; + (0) (On:H
=a (1 - a) (7 (0x:H1 B
((-1) ) +
On: Ha: ))
A
B
ag (DA:Hr + +11 a)
(0g: H B
(1-1)
=a (0x:H "1-)
+ - +1-aa)
H5:
+ +a(1-a) a) (5 (0x: B
+ g (0g:H1
Additionally, we can easily verify that
g0 (0 a + (1 (x) On; H H())) = ag
(On; H())
+ +(1 - a)) (0
(On: Hi))
g (0:H ( A + (1 - a) H (1)
= ag""))
(0: (0:H(()) A
+ aa)" 0 ( (
B
B
Subsequently,
a9 (0A: + (1 aa) (0g: B 1) =ag (0) (ala (1 a1- A )
+ (1 - (1-a) g() (a0, + (1 (1-a) On; H B (1(111)
=/ (a0s ( (1- a1 θg: (H (1-1)
a H (1)
a)
Given the weak additivity for ReLU activation is satisfied for 1 and 0B. then we have
(agi)) (o.:H + (1 (1-a) g (0:: H a (g (al) + (1 a) θ
0 11 11-1) + (1 - a) H )))
(1) (0x:
(t-1) (D:: H + (1 /) (0p; ( nr (a0s + (1 a) 0m
(L-1) + (1 a) H
To conclude, Vl E [L] and Va E [0, 1]. we have
(1-1) oH A + (1 a) H B 9( a0 + (1 a) 0B: ((- + (1 (1 (10)
For the right hand side of Equation
100 recursively, we can have
9 (a0 + (1 - a) 03: + -1-o0 (1 (
0 (a0A + (1- a3:g (1-1) (a0 + (1 a) 03 aH ( 2)
+ (1 - a) H ))
A
= (0 o ((()
+ (1- (1-0)) (l-2) + (1 - a) H (-2)
A
= (9 o ((0 (a0A + (1 (1-a) 0
=f(0 (a0A + (1 (1- 0): X),
14

---

where X denotes the input data matrix.
Recall we denote H(() (2) = =((0) (0; which indicates
af 2 (0x:X)+ + (1 a) f(() (0B: x) f(0 (a0A + (1 a) 0g; X),
and this finishes the proof.
A.3 Derivation of Quadratic Assignment Problem
In this section, we aim to show that minimizing Σ |
(w - plo w p (HI
includes solving Quadratic Assignment Problems (QAPs), known to be NP-hard.
(w p wl! (1) p((-1)
(H - p(( 1
(
(211111 2
arg min T= ==(
H
arg min ==(
( A
(e p w (1) p(t-1)
H((-1) w 1
w H - + pl 1
B H ((-1)) 2
B
arg min (P(P(
[ H
(1 (1-1) - (8)
- p(t-))
H (2- 11-1)
+ plo (2)
( p(t-1)
H (( - p(0
w (2)
H
(e-11) 1)
B
( H( w 0 p H 16-1)
(!! p(e-1) (((-1) p (0) H -
B
B
Consider its first term, i.e.
arg min
W a
((-1))
w p(t-1) (e-1) 2
13
A
n (P())
tr (1x
T
- -H (( p(t-1) ((t-1)
w( ) (WI' (C-1) w
arg min <=((())
A
B
t=1
Σ ir
(
T
w( T
w H (6-1) H (6-1) T p(e-1) w T w ((HH((11) ((1-1)
arg min
((-1))
T
w
w
p(
H (e-1) + H ( p((-1) w T
w p 1) (
A
arg min == (OO))
tt -2H B
T-1 1( T
w
w H + ( p( T w p(t H (-11))
A
A
1r (-2p((-1)
T
((
(( 11-1) 1) H (- + ple-1) w 1000pptt- p H 1) H 11-1)
arg min
A
arg min ==(P(s)
tr (p((-1) w w p(t 1)
H
(t-
H H B
1)
) 2t p((-1)) w w(O H (
B
where in p(e-1) w( T
H (f-1) H
(1-1)
(((--1) T w
T w H H (
22r
is in the form of Koopmans-Beckman''s QAP [14 for each p(t-1) and known as NP-hard. Thus,
solving arg min E/=
W A
(f-1)
w/6
p 1)
1) is to solve L QAPs in parallel.
A
B
x=(p()}
15

---

Similarly, consider the second term, i.e,
P (C) w 4 p(e-1) H(1-1) - w
W 1 H (2-1)) 1)
arg min
==
F
W ( B
p((-1))
H (
arg min -=(
F
tr (H (H"--1 pl:-1) H () w "wb' (wo)p(t-1) p(e-1) H(-1)) -
arg min (P(
t=1
Σ
(
p(t-1) 1 H(-1)) H(6-1)
w(t (8)
T
W W
arg min
B
( w p(e H H
w
W
H
B
B
tr H H(--1) p p(o wi! B
W p(P-11 H ((-11) 2H((-1)
p w w
w ( H (-11)
aig min T= (P
A
B
1r (p(t-)) p(t-1) H((-1) (e-1)) (
(
H (
H ((-1))
arg min T=(P(O)
B
arg min
p(tt-l)
w ( l 1 p(t-1) H ( H
T )
2t
w (
H f H - ))
w H
which also gives rise to Koopmans--Bekkmnn's QAPs. For the last term, i.e,
W W0'HH ((-1)) w plt- H plo p(t-1) H((-1) p(0 w H (1(11))
arg min n =
n
/
F
(w H 1f-1) (0 p " H (W!' p(t-1) H (L-11) - H - ( p!!)
w
arg min (=((())
t=1
which entails solving bi-level matching problems.
Therefore, the objective can be rewritten as the summation of QAPs and bi-level matching problems and cannot be further simplified, which is NP-hard.
B More Experimental Details and Results
B.1 Detailed Experimental Settings
In this section, we introduce the detailed experimental setup. Before delving into details, recall that unless otherwise specified, in this paper we consider models trained on a training set, and then all the investigations are evaluated on test set.
B.1.1 Spawning Method
Multi-Layer Perceptrons on the MNIST Dataset. In accordance with the settings outlined by Ainsworth et al. [], we train multi-layer perceptron networks with three hidden layers, each consisting of 512 units, on the MNIST dataset. We adopt the ReLU activation between layers. Optimization is done with the Adam algorithm and learning rate of 1.2 x 10- The batch size set to 60 and the total number of training epochs is 30. To find the modes that satisfy LMC, we start spawning from a common initialization (0))
16

---

VGG-16 and ResNet-20 on the CIFAR-10 Dataset. In accordance with the settings outlined by Frankle al. [9] we train the VGG-16 architecture 29 and the ResNet-20 architecture [[2] on the CIFAR-10 dataset. Data augmentation techniques include random horizontal flips and random 32 x 32 pixel crops. Optimization is done using SGD with momentum (momentum set to 0.9). A weight decay of 1 x 10-1 is applied. The learning rate is initialized at 0. and is dropped by 10 times at 80 and 120 epochs. The total number of epochs is 160. To find the modes that satisfy LMC, we start spawning after training 5 epochs for both VGG-16 and ResNet-20.
ResNet-50 on the Iiny-ImageNet Dataset. In accordance with the settings outlined by Frankle et al. [9] we train the ResNet-55 architecture [2] on the Tiny-ImageNet dataset. Data augmentation techniques include random horizontal flips and random 32 x 32 pixel crops. Optimization is done using SGD with momentum (momentum set to 0.9). A weight decay of x 10 is applied. The learning rate is set to 0.4 and warmed up for 5 epochs and then is dropped by 10 times at 30, 60 and 80 epochs. The total number of epochs is 90. To find the modes that satisfy LMC, we start spawning after training 14 epochs.
B.1.2 Permutation Method
For the permutation method, we follow the experimental settings of Ainsworth et al. [ strictly, which are described below.
Multi-Layer Perceptrons MNIST and CIFAR-10. Similar to the spawning method, multi-layer perceptron (MLP) networks with three hidden layers, each consisting of 512 units. For MNIST, optimization is performed using Adam with learning rate of 1 x 10-3. For CIFAR-10. optimization is performed using SGD with a learning rate of 0.1. Both activation matching and weight matching are used identify modes that satisfy LMC.
ResNet-20 on CIFAR-10. To achieve LMC, we modify the ResNet-20 architecture by incorporating LayenNorms in place of BatchNorms. Furthermore, we increase the width of ResNet-20 by factor of 32. Data augmentation techniques include random horizontal flips, random 32 x 32 pixel crops. random resizes of the image between 0.8x and 1.2x, and random rotations between =30 The optimization process involves using SGD with momentum (set to 0.9). weight decay regularization term of 5 x 10-4 is applied. A single cosine decay schedule with a linear warm-up is applied, where the learning rate is initialized to x 10-6 and gradually increased to 0. over the course of an epoch, and then a single cosine decay schedule is applied for the remaining training. Only weight matching is used to identify modes that satisfy LMC.
Unlike the spawning method, VGG models are not used in the permutation method due to their inability to achieve LMC. Additionally, Ainsworth et al. [ open-sourced their source code and pre- trained checkpoints, Therefore, we directly use the pre-trained checkpoints provided by Ainsworth
al.
B.2 Verification of LLFC Co-Occuring with LMC
In this section, we provide extensive experimental results to verify that LLFC consistently co-occurs with LMC, and conduct new experiment to demonstrate that the constant c close to I in most cases. Both the spawning method and the permutation method are utilized to obtain linearly connected modes A and O.. As shown in Figures 8 to / 13 and 15 we include experimental results for MLP on the MNIST dataset (spawning method, activation matching, and weight matching), MLP on the CIFAR- 10 dataset (both activation matching and weight matching). VGG-16 16 on the CIFAR-11 10 dataset (spawning method), Re on the CIFAR-10 10 dataset (spawning method and weight matching) and ResNer-50 on the Tiny-ImageNet dataset (spawning method). In particular, in Figure 14, include experimental results of Straight- Trough Estimator (STE) I. STE method tries to learn a permutation with STE that could minimize the loss barrier between mode and the other permuted mode.
To verify the LLFC property on each data point æ; in the test set D, we measure cosine.
coss + (1 a)00;;;;) a;), (0 xi (z,) + (1 -1l-a (() (0 [;)]: We compare this to the baseline cosine similarity cosine B (z,) cos (() (0.; ((0 (0B: In Figures 8 to 15
17

---

MLP on MNIST
1.0
Ep
0.0
FC1
PC2
RC3
RC4
1 co co
1 cosine
cosineo.5
cosine0.5
Eg [ccoee(()))
0.5
0.0
FC1
PC2
IC3
FC4
coefa.25
cocfo.s
cuefazs
Figure 8: Comparison between Ep[[ cosine, (æ:)] and [p[[ cosine (2:)] and demonstration of Ep[1 coef (æ:)]. The spawning method is used to obtain two linearly connected modes 0 a and gg. Results presented for different layers of MLP MNIST dataset, with {0.25, 0.5, 0.75). Standard deviations across the dataset are reported by error bars.
En cosine(x)) Ep[1 cosine(x)))
on CIFARIO
0.5
0.0
Cocr 1-1
Conr 1-2
Cuny 2-1
Conw 2-2
Cour 3-1
Cony 3-2
Coww 3-3
Cour 4-1
T.
I_=
0.0
Cocw 4-2
Con 4-3
Conv 5-1
Conw 5-2
Conr 5-3
PC1
FC2
PC3
cosinex.
cosinuaas
cosinea.50
cosineass
En (x)))
0.5
Cocor
Coor
Conv
2-2
Cour 3-1
Conr 3-2
Corv
Cour
En (caee((k))]
0.5
Coco
Conr
Conv
5-2
Cour 5-3
PC1
FC2
RC3
cocfaas
coefo.s0
coefo.7s
Figure Comparison between Ep[1 cosine, (xx)) and Ep[[ cosinc A, (2;)) and demonstration of Ep 1 coef, (=:)). The spawning method is used to obtain two linearly connected modes 0, and 0B. Results are presented for different layers VGG-16 on the CIFAR-10 dataset, with (0.25, 0.5, 0.75). Standard deviations across the dataset are reported by error bars.
conclude that the values of Ep[1 - cosine, (z;) are close to 0 compared with Ep/1- cosine (x:)), and thus verify our claim.
To show that the constant
close to in most cases, for each data point x; in the test set D. we measure (0;) = (e) (a00 + (1 a)0 æx) |cosine, ( 40 x;) + (1
a) 0 æx) |cosine, (æx) denotes the length of f( x at (On; æx)]]]l where ( (a0 -1-
(1-al (-aa0ss p; mx) projected on (0) .;) )+ (1 a) (() O n; ax)).. ). In Figures to 15 we conclude
that the values of Ep (coef, (x:)) are close to 1 in most cases, and thus verify our claim.
18

---

ResNef20 on CIFAR10
En cosine(x))
1.0
0..5
Block
Block
1-3
Block
Block 2-2
Block
Block3-1
Block
3-ock3-3
cosineaus
cosineg.25
cosineg.75
Ro[1 cosine(G.))
1.0
0.0
Block 1-1
Block 1-2
Block 1-3
Block
B-2
Block
Block3-1
Block32
3-ock
RC
coefa2s
cocfaso
cocfa7s
Figure 10: Comparison between 1 cas and Ep [1 cosine a (x:)) and demonstration of [1 - coef, (a:)]. The spawning method used to obtain two linearly connected modes O and 0 0g. Results are presented for different layers of ResNet-20 on the CIFAR-10 dataset, with E {0.25, 0.5, 0.75J. Standard deviations across the dataset are reported by error bars.
Eb cosine(xx)) Ep cosine(x))
ResNet50 on Tiny- ImageNet
1.0
0.5
Block 1-1
Block 1-2
Black 1-3
Block 2-1
Block 2-2
Block
Block
Block
1.0
0.5
Block 3-2
Block
Block 3-4
Block 3-5
Block 3-6
Block
Block
Block 4-3
сosine
cosine
cosine
cosine,.s
Ep (coet(x))] [x(c
0.5
Block
Block
Block
Block 2-1
Block 2-2
Block
Block
Block 3-1
0.5
Block
Block
Block
Block
Block
Block 4-1
Block
Block
coefo.2
coefo.50
coefo75
Figure 11: Comparison between (o]1-cosine,.a:)) and Ep -ls--coinne,, A, (xx)) and demonstration of [1 - coef, (æ:)). The spawning method used to obtain two linearly connected modes O and 0g. Results are presented for different layers of ResNet-50 E (0.25, 0.5, 0.75). Standard deviations across the dataset are reported by error bars.
the Tiny-ImageNet dataset, with
B.3 Verification of Commutativity
In this section, we provide more experimental results on various datasets and model architectures to verify the commutativity property for modes that satisfy LLFC. As shown Figures 16 [8] we include more experiments results for VGG-16 on the CIFAR-10 dataset (spawning method). MLP on the MNIST dataset (activation matching) and MI.P on the CIFAR-10 10 dataset (both activation matching and weight matching).
19

---

MLP MNIST (Adivation Matching)
MLP on MNIST (Weight Matching)
Boll cosine(())
Eall cosine(x,)
0.0
FC1
FC2
RC3
RC4
RC1
PC2
FC3
FC4
cosines.
cosinco.2
cosineo.50
cosine
[eneel(()))
1.0
En(coef(x.))
0.5
0.5
FC1
coefo.s
cucfo.o
coefars
Figure 12: Comparison between Ep 1all-osionn cosine, (x;)] and Ep 11- -co-cosin,,, A.B (æ;)] and demonstration of Ep coefo (a:)). The activation matching and the weight matching are used to obtain two linearly connected modes 0 and 0g. Results are presented for different layers of MLP on the MNIST dataset, with E {0.25, 0.5, 0.75}. Standard deviations across the dataset reported by error bars.
Boll cosine(x)]
MLP on CIFARIO (Adivation Matching)
Egll cosine(x))
MLP CIBAR10 (Weight Matching)
1.0
1.0
0.5
0.5
0.0
0.0
RC2
RC3
FC4
FC1
RC2
RC3
cosine
cosine
-cosine so
cosine..75
1.0
(o(coe((x))]
1.0
0.0
0.0
FC1
FC2
RC3
FC4
PC1
PC2
FC3
FC4
coefazs
coefy.s0
coefa.5
Figure 13: Comparison between - and Ep - cosinc A, e (x:) and demonstration of Ep coef, (æ:)). The activation matching and the weight matching are used to obtain two linearly connected modes O and On Results are presented for different layers of MLP the CIFAR-10 dataset, with a E {0.25, 0.5, 0.75J. Standard deviations across the dataset are reported by error bars.
cosine,,
To verify the commutativity generally holds for modes that satisfy LLFC, for test set we compute Distcom dist (vec( w H
(f- w (f) H B
(f-1)) vee (t) H
(f-1)
w
(1)
B H
(f
Fur-
thermore, we compare Distpom with Distw dist (vec(Wil)) (0) vec W and Dist
dist (ve((H(((-1)) vec(H! (- respectively. In Figures 16 pared with Distw and Distu, confirming the commutativity condition.
18 Distcom is negligible com-
Furthermore, we add baselines of models that are linearly connected further validate the commutativity condition. In Figure [9] we include experimental results for ResNet-20 on CIFAR-10 dataset (both spawning and weight matching method). Specifically, we measure Dist LMC of two linearly connected modes and Dist LMC of two independently trained modes. In Figure T9 the values of Distcom.LMC are negligible compared with Distcom,not LMC' which confirms the commutativity condition.
'We also conduct experiments on CNNs. For a Conv layer, the forward propagation will be denoted as W WH similar to linear layer.
20

---

ML MNIST (STE)
ML CIFARIO (STE)
Boll cx)ine(x))
Epl1 - cosine(ar))
1.0
1,0
FC1
FC3
RC4
FC1
RC2
FC3
FC4
cosine
cosinCo.2
cosineo.50
cosincp.75
Ep ccoef(x)))
En(coof((x))
0.5
0.0 RC1
0.0
RC4
RC2
RC3
coefa.5
coefo.50
coefo7s
Figure 14: Comparison between [p[1-cosine,/e and Ep cosine A, (x:)] and demonstration of [1 coef, (a::)] The Straight-Through Estimator (STE) M used obtain two linearly
connected modes A and gg. Results are presented for different layers of MLP on both MNIST and
CIFAR-10 dataset, with {0.25, 0.5, 0.75}. Standard deviations across the dataset are reported by еπгог bars
ResNet20 (32 x) on CIBARIO (Weight Matching)
En[1 - costne(x.)]
1.0
Block
Block 1-2
Block 1-3
Block 2-1
Block
Block
Block
Block
Block33
FC
cosine
cosine,as
-cosine.50
-cosinep.s
Bo[1 - coslne(k,)]
0.5
Block
Block 1-2
1-3
Block
Block
Block
Block3-1
Block
Block33
PC
coefaas
coefa.so
coefass
Figure 15: Comparison between Ep[1 Bp[1-cosines,
(x:)nee and Ep cosine B (x;)) and demonstration
of Ep[1 coef, (a:;). The weight matching is used to obtain two linearly connected modes 0 a and
On- Results presented for different layers of ResNet-20 (32x) the CIFAR-10 dataset, with E (0.25, 0.5, 0.75). Standard deviations across the dataset are reported by error bars.
VGG16 CIFAR1O
15
Diebarze
Discan
Distane
Disauree
Conv 1-2
Cony 2-1
Cony 2-2
Conv 3-1
Conv 3-2
Conv 3-3
1.00
Dislanee
Distano
Distance
Distaree
Distave
Com 4-1
Corw 4-2
Cory 4-3
Conr 5-1
Conw 5-2
Conw 5-3
Distcom
Distw
Disty
Figure 16: Comparison of Distcom Distw: and Disty- The spawning method used to obtain two modes that satisfy LLFC, A and 0g. The results are presented for different layers of VGG-16 on the CIFAR-10 dataset.
21

---

MLP MNIST (Acivation Matching)
Distaroe
FC2
FC3
FC4
Dist.com
Distay
Disty
Figure 17: Comparison of Distcom Distw, and Disty- The activation matching is used to obtain two modes that satisfy LLFC, A and Bp. The results are presented for different layers of MLP on the MNIST dataset.
MLP om CIEARIO (Activation Matching)
MLP on CIFAR1O (Weight Matching)
Distano
Distance
Desunne
Destonee
9.5
Distangs
Distanz
FC2
RC3
FC 4
RC
FC3
FC4
Dist.om
Distw
Disty
Figure 18: Comparison of Distcom Distw, and Distæ- Both the activation matching and weight matching are used to obtain two modes that satisfy LLFC, A and 0n. The results are presented for different layers of MLP on the CIFARIO dataset.
CIEARIO (Spaweing)
ReaNar20 CIPARIO Marting)
Detaace
Dataanee
Block 23
Block
Bloc332
Block 3-
FC
Block 2-3
Bloc3-1
Blok332
Bloc333
Distoon
Bisteon
Figure 19: Comparison between Distcom, and Dist.om, LMC- Both the spawning and permu- tation methods used to obtain two linearly connected modes.
Layer
FC2
FC
2.69
2.11
1.92
Table Error rates (%) of stitched MLP on the MNIST test set. The model stitching is employed in different layers. The spawning method is used to obtain two neural networks that satisfy LLFC, i.e., A and B. Error rates (%) of A and B are 1.9 and 1.77, respectively.
Notably, the experiments not conducted on the first Conv/Linear layer of the model because the commutativity condition is naturally satisfied for the first layer where H (!) X where X is the input data matrix.
B.4 Experiments on Model Stitching
Model stitching 19 3 is commonly employed to analyze neural networks' internal representations. Let A and represent neural networks with identical architectures. Given loss function L, model stitching involves finding stitching layer (e.g., a linear 1 1 convolutional layer) such that the minimization C( o s o Age) is achieved. Here, Byt denotes the mapping from the activations of the L-th layer of network B to the final output, Age denotes the mapping from the input to the activations of the L-th layer network A, and represents function composition.
In this section, we explore a stronger form of model stitching. Specifically, given two neural networks and that satisfy LLFC, we evaluate the arruacy of By Ace over test set D without finding stitching layer, i.e., (or As shown in Tables [ to we include experimental results for MI.P the MNIST dataset, VGG-16 on CIFAR-10 10 the dataset and ResNet-20 on the CIFAR-10 dataset. Only the spawning method is utilized to find modes that satisfy LLFC. The results depicted
22

---

Layer
Conv 1-1
Conv 1-2
Conv 2-1
Conv 2-2
Conv
7.2
8.43
8.39
9.91
11.84
Layer
Conv 3-2
Conv 3-3
Conv 4-1
Conv 4-2
Conv 4-3
9.55
8.22
7.61
6.99
7.05
Layer Conv 5-1
Cony 5-2
Conv 5-3
RC
FC22
6,91
6.88
6.88
7.07
6.92
Table 2: Error rates (%) of stitched VGG- on the CIFAR-I0 test set. The model stitching
employed in different layers. The spawning method is used obtain two neural networks that satisfy LLFC, i.., A and B. Error rates (%) of and B are 6.87 and 7.1, respectively.
Layer
Block 1-1
Block 1-2
Block 1-3
Block 2-1
Block
Block 2-3
Block 3-1
Block 3-2
Block 3-3
10.88
10.57
13.35
10.64
10.74
10.55
12.27
11.8
8.99
Table 3: Error rates (%) of stitched ResNet-20 on the CIFAR-10 10 test set. The model stitching is employed in different layers. The spawning method is used obtain two neural networks that satisfy LLFC, i.e. and B. Error rates (%) of and B are 8.69 and 8.58, respectively.
MLP on CIRARIO
valouss
Singnaavaauss
ACE
203
Index of
Index of
Index of
PC
FC2
FC3
ResNet20 (32 x) on CIFARIO
Singullovee voluus
Gongalaa werelaavaaal valuuss
Suxgala voluuus voluss
Sungabar Va valuuns
Singalaar volluess
Singlla wongulaavauuue vuluus
3.8
5
Inckoc of s
Index of
Irdex of
Incex of
Index ofs
Irdex of
Block3-1 Com
Block 3-1 Com 2
Block3 Conv
Block332
Block 3-3
Block 3o
Figure 20: Singular values of weight matrix ((()) of t-th layer of 0 in a descending order. Here, can be used achieve LMC with weight matching. The results presented for different layers of various model architectures and datasets.
in Tables 3 demonstrate that the error rates of the stitched model the test set closely resemble the error rates of the original models and B, regardless the dataset model architecture. This observation suggests that models that satisfy LLFC encode similar information, which can be decoded across different models. Subsequently, the experiments of model stitching provides new insights
towards the commutativity property, i.e, vl E [L]. w (
H (f-1)
w(() H B
B.5 Discussion on Git Re-basin
In this section, we investigate the ability of permutation methods to achieve LMC. While we have interpreted the activation matching and weight matching methods proposed by Ainsworth et al. [I] as guaranteeing the commutativity property, we have yet to address why permutation methods can ensure the satisfaction of this property. Thus, in order to delve into the capability of permutation methods, we must address the question of why these methods are capable ensuring the satisfaction of the commutativity property.
Low-rank model weights and activations contribute to ensure the commutativity property. We now consider stronger form of the commutativity property, where given two modes O and 0 and
23

---

MLPon on CIFAR1O
Langnaavvlueee
Singdarvaauue
2000
Singulor
Ircex
Index
RC1
RC2
FC3
Figure 21: Singular values of post-activations ( of 2-th layer of over the whole test set D in a descending order. Here, 0 can be used to achieve LMC with activation matching. The results are presented for different layers of MLP on the CIFAR-10 dataset.
ResNet20 (32 x) on CIFARIO
1200
LX04
Srank(W>
E09
300
400
grank (W7
Srank
Block 3-1 Convl
Block 3-1 Conv2
Block 3-2 Conv1
Block 3-2 Com2
Block 3-3 Conv1
Block 3-3 Comv2
Here, W denotes the weight matrix of the E-th layer of the model 0r in the terminal phase of training. Similarly, w denotes the weight matrix of the f-th layer of the model 02 in the early stage of training (1 epoch indeed). Also, the stable rank can be calculated as srank (W) for different layers of ResNet-20 (32x) on the CIFAR-10 dataset.
Figure 22: Comparion between the stable rank srank W and srank (W((). srank(W(())
|W
The results are presented TW
dataset DD, we have:
vl [L]. W H (--) w/ (0
( (L-1) w
H (C-1)
w H (E-1)
Thus, to satisfy the commutativity property for a given layer E, we can employ the permutation method to find a permutation matrix p!e-1) such that:
(f- = 0 p w ( H ((-))) p((-1) H
H B (H-1)) - H x =0, 0.
In homogeneous linear system W 0, low-rank matrix W allows for larger solution space for X. Therefore, if the ranks of ((( matrix ((-11) that satisfies the commutativity property. Similarly, if we consider another form of commutativity property:
A and ( B are low, it becomes easier to find a permutation
vl E [Z]. w H 1)
H ( (e-1) A w H ((-1) w H 1
Then, to ensure the commutativity property, we need find ((t-1) and p(() such that
W (6)
((())))))))))))) p(e-1) H((-1)
0n (p( ww p(e-1) w() (( 1 =.
Then, if the ranks of H and H (t-1) are low, is easier to find the permutation matrices satisfy the condition. In real scenarios, both model weights (see Figure 20) and activations (see Figure 21) approximately low-rank, which helps the permutation methods satisfy the commutativity property.
Additionally, Ainsworth al. M mentioned two instances where permutation methods can fail: models with insufficient width and models in the early stages of training. In both cases, the model weights often fail to satisfy the low-rank model weight condition. In the first scenario, when the model lacks sufficient width, meaning that the dimension of the weight matrix approaches the rank of the weight matrix, the low-rank condition may not be met. For example, compared the singular values of ResNer-20 (32x) (see Figure 20) with singular values of ResNet-20 (1x) (see ??), it is evident that in the wider architecture, the proportion of salient singular values smaller. the second scenario, during the initial stages of training, the weight matrices resemble random matrices and may not
24

---

exhibit low-rank characteristics. For example, as shown in Figure 22 the stable ranks of weight matrices of the model after convergence are significantly smaller than those of the model in the early stage of training. Consequently, permutation methods may struggle to find suitable permutations that fulfill the commutativity property, resulting in the inability to obtain modes that satisfy LMC.
25


Reviews:
[{"review_id": "0rVY6WYsKD", "reviewer": "Reviewer_6KFq", "summary": "This paper presents a special case of Linear Mode Connectivity (LMC) denoted as Layerwise Linear Feature Connectivity (LLFC). Whereas two trained neural networks present LMC if a convex combination of their parameters produce a neural network with similar training loss and accuracy, two trained neural networks present LLFC if a convex combination of their features in every layer also produces a neural network with similar features. Although being a special case, the authors find out that LMC and LLFC co-occur very often. Moreover, the authors characterize conditions that jointly imply LLFC between two ReLU networks.\n\n*****\n\nFollowing the authors' response, I am updating my score from 6 to 7.", "questions": "The introduction attributes the first observation of LMC to [8], but Section 2 attributes it to earlier work [21]. \n\nDefinition 2 needs rework because it is clearly invalid if $\\alpha=0$ or $\\alpha=1$ unless $c=1$. In fact, this is not even how you measure it in Section 4, since you use cosine similarity instead. I believe that the use of a tolerance factor  $\\epsilon$ like you did in Line 220 would be more adequate in this definition.\n\nWould it make sense to assume that Definition 3 has some connection with stable neurons in ReLU networks, as in the following papers: https://arxiv.org/abs/1711.07356 & https://arxiv.org/abs/2102.07804 ?\n\nIt seems to me that Condition 1 was inspired by spawning and Condition 2 by permutation, even though they are not worded in that way. Is that a valid way to interpret them?\n\nI would like to hear the thoughts of the authors about what I described in Weaknesses.", "limitations": "I could not identify a discussion about limitations in the paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The writing of the paper is very clear and the experiments are distributed in such a way along the text that it is easy to follow along.\n\nMoreover, I find the study of how this conditions may emerge in ReLU networks particularly relevant, since in that simpler setting it is easier to understand what is going on.", "weaknesses": "The authors sell the idea of LLFC in a very positive tone, which is actually quite common in papers, but I cannot help but wonder about the following: \n\nTwo neural networks presenting LLFC would have very similar parameters, and the fact that LLFC co-occurs with LMC implies that LMC is observed due to the similarity between trained neural networks - either because the first epochs of training determined what model would be ultimately obtained; or because there are relatively few optimal neural networks upon permutation.\n\nHowever, even under this interpretation, I believe that this study helps demystifying LMC if it turns out that LMC rarely occur without LLFC.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Concerns about the trivial case that two Neural Networks (NNs) emerging LLFC share the similar weights. “ I cannot help but wonder about the following: Two neural networks presenting LLFC would have very similar parameters, and the fact that LLFC co-occurs with LMC implies that LMC is observed due to the similarity between trained neural networks”**\n\n**A1**: Thank you for your great question. In fact, our experiments have already ruled out the trivial case that two NNs share similar weights. \n\nFirst, we directly evaluated the difference between the weights of two NNs that yield LLFC, namely $\\text{Dist}_W = \\text{dist}\\left(\\text{vec}({\\boldsymbol{W}}_A^{(\\ell)}), \\text{vec}({\\boldsymbol{W}}_B^{(\\ell)})\\right)$, where $\\text{dist}(\\boldsymbol{x}, \\boldsymbol{y}) := \\|\\boldsymbol{x} - \\boldsymbol{y}\\|^2 / (\\| \\boldsymbol{x}\\| \\cdot \\|\\boldsymbol{y}\\|)$. In Figure 5 (main paper), the values of $\\text{Dist}\\_W$ are usually within 0.5~1.5 while the values of $\\text{Dist}\\_{com}$ are close to zero. \n\nSecond, we calculated the cosine similarity between the features of two linearly connected NNs, namely, ${\\rm cosine}\\_{A, B}(\\boldsymbol{x}_i)$. If the weights of two NNs are similar, their features should display similarity as well. Nevertheless, in Figure 2 and 3 (main paper), the value of $\\mathbb{E}\\_{\\mathcal{D}}[1-{\\rm cosine}\\_{A, B}(\\boldsymbol{x}_i)]$ could reach its maximum at around 0.75.  \n\nConsequently, we can confidently dismiss the trivial case that two NNs emerging LLFC share similar weights.\n\n**Q2: Whether LMC rarely occur without LLFC. “However, even under this interpretation, I believe that this study helps demystifying LMC if it turns out that LMC rarely occur without LLFC.”**\n\n**A2**: Thank you for acknowledging our contribution. We would like to clarify that we did not find any instances where two NNs exhibit LMC but not LLFC. We conducted extensive experiments across diverse datasets, network architectures, and various layers within the network (Sec 4 & Appendix D.2). Therefore, we believe LLFC is a more fundamental phenomenon that helps demystify LMC.\n\n**Q3: A reference problem. “The introduction attributes the first observation of LMC to [8], but Section 2 attributes it to earlier work [21].”**\n\n**A3**: Thank you for pointing out this problem.  In fact, [8] references [21] and attributes the initial observation to [21]. However, [8] was the first to formally define and thoroughly investigate the LMC problem. Thank you for your careful review, we will later clarify this point in the revised version of our paper.\n\n**Q4: Concerns about Definition 2. “Definition 2 needs rework because it is clearly invalid if $\\alpha =0$ or $\\alpha =1$ unless $c=1$.”**\n\n**A4**: Thank you for your careful review again. In Definition 2, $c$ is allowed to depend on the interpolation parameter $\\alpha$, i.e., \"$\\forall \\alpha \\in [0, 1], \\exists c > 0$\". When $\\alpha = 0$ or $\\alpha = 1$, $c=1$ will satisfy the condition. Consequently, the definition remains valid even at the boundary cases of $\\alpha = 0$ or $\\alpha = 1$. For other values of $\\alpha$, $c$ can be different from $1$. Thus, using cosine similarity to verify LLFC (Definition 2) is appropriate.\n\n**Q5: Where Definition 3 has some connection with stable neurons in ReLU networks. “Would it make sense to assume that Definition 3 has some connection with stable neurons in ReLU networks…?”**\n\n**A5**: Thank you for your question. Regarding the two papers you referenced, stable neuron is defined as one whose output is the constant value zero ($y=0$) or the pre-activation output ($y=x$) on all inputs, which is a property concerning a single network. On the other hand, Definition 3 concerns a relation between two networks. Therefore, these two properties are orthogonal to each other, and their connection is not clear. That said, both properties describe interesting linearity phenomena in ReLU networks, and we will make a note of this in the paper.\n\n**Q6: Question about the inspiration and interpretation of Condition 1 and 2. “It seems to me that Condition 1 was inspired by spawning and Condition 2 by permutation, even though they are not worded in that way. Is that a valid way to interpret them?”**\n\n**A6**:  We are happy to explain what inspired Conditions 1&2. In fact, they were not inspired from spawning or permutation methods but stemmed from the derivation process of Theorem 1. Through this, we identified that the two conditions facilitate the derivation of the feature connectivity across layers. Subsequently, we discovered the connection between Condition 2 and both spawning and permutation methods. This intriguing connection leads us to the conjecture that both spawning method and permutation method essentially contribute to the fulfillment of LLFC.\n\n**Q8: Ask for more discussion about the limitations of this paper. “I could not identify a discussion about limitations in the paper.”**\n\n**A8**: We are happy to discuss the limitations of this paper. Please kindly refer to the global response for a list of limitations, which we will add to the paper."}, {"review_id": "IFBcHkYcSL", "reviewer": "Reviewer_Dt29", "summary": "In this work, the property of Layerwise Linear Feature Connectivity (LLFC) of neural network representations is introduced, which is a stronger generalization of linear mode connectivity (LMC). They show that LLFC often occurs when LMC does. Moreover, they give a possible mechanism by which LLFC may occur (ReLU activation additivity and weight commutatitivty), and provide evidence that LLFC often does occur by this mechanism. Finally, they reinterpret spawning and permutation-finding based methods for LMC as promoting commutativity.\n", "questions": "Typos:\n1. In 5.2, (i) should have $W_{B, \\mathrm{pri}}^{(l)} H^{(l-1)}_A + W_{A, \\mathrm{pri}}^{(l)} H_B^{(l-1)}$.\n2. 5.3: \"conprehensive\" -> \"comprehensive\"\n", "limitations": "I do not see much discussion of limitations, besides on the hardness of exactly minimizing the commutativity property objective. Perhaps you could state that the empirical evidence is limited to image classification.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Introduction exposition is quite good.\n2. The relationship between spawning / permutation-finding methods and commutatitivity is really interesting and insightful. The fact that both the weight-matching and activation-matching losses can be found in the commutativity condition is quite neat, as (to the best of my knowledge) these two methods were proposed as heuristics. The discussion of weight matrix rank in Appendix E is also insightful.\n3. Figure 5 on the empirical measurement of commutativity is very nice, especially because the two baseline numbers (weight distance and activation distance) are so large, so commutatitivity really seems to be a good thing to look at here.\n4. Although LLFC is stronger than LMC, in some sense it may make the study of LMC easier, because you have more to look at (features in each layer and the two sufficient conditions you give, not just loss of a whole neural net).\n\n", "weaknesses": "1. You don't really cover the last method (straight-through estimator) from Git Re-Basin, even though it works the best in many cases. Perhaps you should note this.\n2. Although these experimental setups are standard, it would be good to see to what extent this holds beyond these few architectures for image classification.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Ask for additional experiments to cover the straight-through estimator method from Git Re-basin [1]. “You don't really cover the last method (straight-through estimator) from Git Re-Basin, even though it works the best in many cases. Perhaps you should note this.”**\n\n**A1**: We greatly appreciate your suggestions. We follow your suggestions and conduct new experiments to cover the Straight-Through Estimator (STE) method. \n\nOur experiments follow the same settings as in Ainsworth et al. [1], which applied STE method on MLPs trained on MNIST and CIFAR-10 datasets. Correspondingly, in congruence with the experiments in Section 4 of the main paper, we measure both ${\\rm cosine}\\_{\\alpha}(\\boldsymbol{x}_i)$ and ${\\rm cosine}\\_{A, B}(\\boldsymbol{x}_i)$ for each data point $\\boldsymbol{x}_i$ in the test set $\\mathcal{D}$. In Figure 1 (global response), the values of $\\mathbb{E}\\_{\\mathcal{D}}[1-{\\rm cosine}\\_{\\alpha}(\\boldsymbol{x}_i)]$ are close to zero compared to $\\mathbb{E}\\_{\\mathcal{D}}[1-{\\rm cosine}\\_{A,B}(\\boldsymbol{x}_i)]$ across different layers, datasets, and different values of $\\alpha$, which further convincingly verifies that LLFC consistently co-occurs with LMC. Please kindly refer to the global response for more experimental details.\n\n**Q2: Suggestions for empirical evidence beyond image classification. “Although these experimental setups are standard, it would be good to see to what extent this holds beyond these few architectures for image classification.”** \n\n**A2**: Thank you for your suggestions. We acknowledge that our experiments follow the standard setups in the LMC literature. We will leave the exploration beyond image classification as future directions. We have included this in the list of limitations (see global response).\n\n**Q3: Typos.**\n\n**A3**: Thank you for spotting the typos! We will fix them.\n\n**Q4: Ask for more discussion about the limitation of this paper. “I do not see much discussion of limitations, besides on the hardness of exactly minimizing the commutativity property objective. Perhaps you could state that the empirical evidence is limited to image classification.”**\n\n**A4**: Thank you for the suggestion. Please refer to the global response for a list of limitations, which we will add to the paper."}, {"review_id": "EBwkbXiYmS", "reviewer": "Reviewer_ZZ9i", "summary": "This paper introduces Layerwise Linear Feature Connectivity (LLFC). Compared to the better known linear mode connectivity (LMC), which states that networks trained by SGD are linearly connected modulo permutation, LLFC suggests that the feature maps of every layer is connected. As shown in the paper, LLFC is a strictly stronger property than LMC. Empirical results on a range of architectures (ResNet-20, VGG-16, MLP) and datasets (MNIST, CIFAR-10) suggest that LLFC co-occurs with LMC. The authors then identify two conditions that collectively imply LLFC: weak additivity, which requires ReLU to behave like a linear activation on two modes, and commutativity, which requires the next-layer linear transformations applied to the internal features of two networks can be interchanged. They verify that these two properties holds for modes that satisfy LLFC empirically. Finally, the authors show that two common methods to obtain linearly connected modes, the spawning method and the permutation method, both promote the commutativity property, which explains their effectiveness.\n", "questions": "- In the definition of LLFC, there is a scaling factor $c$ that is not predicted by theorem 1. According to the authors, this inconsistency can be attributed to the accumulation of errors in the two conditions (line 263). Why does the accumulation of errors result in a scaling difference instead of another type of modification such as an additive term? \n- Does spawning or permutation have any impact on the sparsity of pre-activation $\\tilde{H}^{(l)}$ and the weak additivity condition? \n- Is condition 1+2 the only way to guarantee LLFC? In particular, identifying other sets of sufficient conditions for LLFC may lead to new permutation methods that find linearly connected modes, other than weight matching or activation matching.\n- The experiments use SGD with momentum in training. Does the choice of optimization method affect the occurrence of LLFC? \n", "limitations": "The authors adequately addressed the limitations. There are no potential negative societal impacts of this work.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper reveals a novel and more general notion of linear mode connectivity, which is an interesting phenomenon that attracts recent attention in the ML community. The authors provide precise definitions and a set of sufficient conditions for LLFC. Their observation is novel and advances the understanding of the origin of linear connectivity. \n- Experiments are sound and provide useful insights. In particular, empirical results support the occurrence of LLFC and validates that the two conditions for LLFC approximately hold in common settings. \n- Through dissecting the conditions required for LLFC, this paper also explains why the spawning method and permutation method produces LMC. Since theoretical results are sparse in explaining the origin of LMC, this paper’s contribution on the topic is significant. \n- The writing is clear and well organized. \n", "weaknesses": "- The analysis for the cause of LLFC is limited to ReLU activation. Extending the weak additivity condition to different activations does not seem straightforward. To demonstrate the prevalence of LLFC, it might help to include experiments on neural networks with different activations. \n- As pointed out by the authors and shown in figure 4 and 6, weak additivity for ReLU activations and commutativity only approximately hold in real neural networks. Hence Theorem 1 describes conditions for perfect LLFC in an idealistic setting. While this result is significant and novel, a more careful treatment of the approximated version could make the theoretical contribution stronger. \n- There are some gaps between theory and empirical observation. Specifically, the following results are not well explained: (a) Why are the pre-activations sparse? Is this universal to all architectures? (b) Why do the modes obtained by the spawning method share similar principal directions of model weights in each layer? While these are empirically verified in the paper, it would be helpful to point the readers to existing theoretical analysis, if available. \n- While LLFC is an interesting observation and provides insights on LMC, there seems to be few applications that leverages LLFC. The idea of averaging features mentioned in the conclusion is interesting, but it is not clear how to implement feature averaging and under what situations this would be beneficial.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: “The analysis for the cause of LLFC is limited to ReLU activation…To demonstrate the prevalence of LLFC, it might help to include experiments on neural networks with different activations.”**\n\n**A1**: We note that ReLU is the activation used in standard architectures including ResNet, VGG, etc., and is used in all previous work on LMC. An analysis based on ReLU is already highly informative given its wide adoption. In fact, we did some preliminary experiments which showed that LMC is difficult to achieve for some other activation functions. We leave the exploration beyond the ReLU activation as a future direction.\n\n**Q2: “While this result is significant and novel…the approximated version could make the theoretical contribution stronger.”**\n\n**A2**: Thank you for appreciating the significance and novelty of Thm 1. We agree that an approximate version could make the theoretical contribution stronger. We currently adopt the exact formulation for ease of presentation, as we believe the concise formulation already conveys the main ideas and is easier to understand and build upon. \n\nWe remain open to add a careful approximate version of the theorem to the paper. Please kindly refer to our global response for more discussion on the limitation of our work.\n\n**Q3: “There are some gaps between theory and empirical observation…(a) Why are the pre-activations sparse?…(b) Why do the modes obtained by the spawning method share similar principal directions of model weights in each layer?…it would be helpful to point the readers to existing theoretical analysis, if available”**\n\n**A3**: We agree that (a) and (b) are intriguing phenomena, and theoretical explanations would be valuable. However, we are not aware of any existing theoretical analysis that's directly relevant. There are some empirical studies that bear some degree of relevance, e.g., [cite 2] empirically investigate the sparsity of the activations; [cite 1] studies the evolving statistics of model weights during the early phase of training. We agree that theoretical studies of these questions are important future directions.\n\n[cite 1] Jonathan Frankle, David J. Schwab, and Ari S. Morcos. The early phase of neural network training. In International Conference on Learning Representations, 2020.\n\n[cite 2] Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. J. Mach. Learn. Res., 22(1), jan 2021. ISSN 1532-4435\n\n**Q4: “…there seems to be few applications that leverages LLFC…”**\n\n**A4**: The goal of this paper is to unveil nontrivial phenomena that offer elucidating insights into the fundamental mechanisms of deep learning. Akin to how LMC served as inspiration for applications like model soup [31], LLFC could potentially inspire applications such as feature averaging. We leave this prospect as a future direction, opting not to explore it within the scope of a single paper.\n\n**Q5: “In the definition of LLFC, there is a scaling factor $c$ that is not predicted by theorem 1…Why does the accumulation of errors result in a scaling difference instead of another type of modification such as an additive term?”**\n\n**A5**: Thank you for the great question. First of all, in most cases, we empirically observe that $c$ is close to 1 (See Appendix D.2), as predicted by Theorem 1. In other cases, we find that employing a scaling factor enables a much better description of the practical behavior than an additive error term. We will include this discussion in the paper.\n\n**Q6: “Does spawning or permutation have any impact on the sparsity of pre-activation $\\tilde{\\boldsymbol{H}}^{(\\ell)}$ and the weak additivity condition?”**\n\n**A6**: Both the spawning and permutation methods do affect the weak additivity condition. In Figure 3 (global response), we show that the values of $\\text{Dist}\\_{\\sigma}(\\tilde{\\boldsymbol{h}}\\_{i, A}, \\tilde{\\boldsymbol{h}}\\_{i, B})$ (with spawning or permutation) are negligible compared to $\\text{Dist}\\_{\\sigma}(\\tilde{\\boldsymbol{h}}\\_{i, C}, \\tilde{\\boldsymbol{h}}\\_{i, D})$ (independently trained networks). Please kindly refer to the global response for more details. On the other hand, pre-activation sparsity is not affected because it is only a property of a single network.\n\n**Q7: “Is condition 1+2 the only way to guarantee LLFC? In particular, identifying other sets of sufficient conditions for LLFC may lead to new permutation methods…other than weight matching or activation matching.”**\n\n**A7**: A very interesting question. Based on our theoretical and empirical results, we believe that LLFC is strongly related to Conditions 1+2. On the other hand, while Conditions 1+2 are very insightful for our understanding of LLFC and LMC, we agree that if there exist other sufficient conditions, they might be helpful for designing new permutation methods.\n\n**Q8: “The experiments use SGD with momentum in training. Does the choice of optimization method affect the occurrence of LLFC?”**\n\n**A8**: Thank you for your question. Indeed, we have carried out experiments employing the Adam optimizer for MLPs trained on the MNIST dataset. Consequently, we believe that the choice of optimization methods does not affect the occurrence of LLFC. To further verify, we conduct additional experiments entailing the training of ResNet-20 on the CIFAR-10 dataset with Adam optimizer, as shown in Figure 1 (global response). We kindly direct you to the global response section for more experimental details."}, {"review_id": "vaSC34PolT", "reviewer": "Reviewer_4GwW", "summary": "The work identifies layerwise  linear feature connectivity (LLFC) and proves LLFC is sufficient for linear mode connectivity (LMC) between neural networks. Experimental evidence using two methods for finding LMC networks (spawning and permutation) finds that LMC and LLFC co-occur in a variety of models and datasets. Two conditions (weak additivity of ReLU and commutativity) are identified as sufficient for LLFC, and experimental evidence is presented for these conditions occuring in LMC networks.", "questions": "- Section 5.2: how  does similarity of $U_A$ and $U_B$ imply commutativity when $W = U \\Sigma V$ also depends on $V$, which could differ significantly between two networks? The connection between figure 6 and the equation in lines 285-286 is unclear.\n- How much would solving the quadratic assignment problem improve over existing algorithms? Given that activation similarity is closely correlated with weight similarity, it is not obvious whether minimizing the QAP will lead to significant improvements in permutation alignment versus aligning weights and/or alignments, which is equivalent to minimizing one or both sides of equation 9 separately.\n- Weak additivity seems to rely heavily on the ReLU function. Does LLFC occur for other activation functions?\n- Do networks exist where LMC holds but not LLFC? If so, how can one find such networks and how common are they?", "limitations": "Overall the findings could be explored in greater depth and better related to observable phenomena. While LLFC is shown to imply LMC, the reverse is not shown or refuted. For spawned LMC networks, it is not clear whether LLFC is a byproduct of averaging, how LLFC relates to the onset of LMC at an early point in training, or how LLFC can be increased/decreased. For permuted LMC networks, current alignment algorithms (e.g. Ainsworth et al. 2022, Singh & Jaggi 2020) are already explicitly optimizing for LLFC (via weight or activation alignment), and thus the findings have limited explanatory power.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper unifies two lines of research into LMC (spawning versus permutation methods) under a common mathematical framework. The notion of commutativity is an interesting derivation and not as obvious as it seems at first glance. The paper is well written and easy to follow, with clear definitions. If established comprehensively, LLFC would be a significantly stronger condition than LMC, and would have far-reaching implications.", "weaknesses": "Main issues:\n- Current alignment algorithms for permuting LMC networks already directly optimize for LLFC (section 5.3), so the finding that LLFC implies LMC (Lemma 1) is formalizing a well-established phenomenon which is somewhat obvious.\n- As a result, LLFC implying LMC is somehow less interesting than LMC implying LLFC, but the latter direction is not explored. Only co-occurence is observed experimentally, making the direction of causation unclear. In particular, if LMC is found to imply LLFC, the reasons are likely to be extremely informative.\n- For spawning LMC networks, it is unclear how the early training epochs contribute to enabling LMC. The experiments do not consider the evolution of LMC networks through training time. Furthermore, the connection between figure 6 and the equation in lines 285-286 is unclear (see questions below).\n\nAdditionally, the experimental evidence lacks baselines for comparison. In general, a fair comparison should include interpolated non-LMC networks.\n- Figure 2 and 3: no cosine similarity for interpolated networks that are not linearly connected. Since interpolating between any vectors (including random ones) increases their cosine similarity considerably, it is not clear that the increase in similarity is due to LLFC and not averaging.\n- Figure 4: no sparsity baseline for random networks. Currently it is unclear what is considered \"small\" in line 232.\n- Figure 5: no commutativity distance for non-LMC networks", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: “Current alignment algorithms for permuting LMC networks already directly optimize for LLFC (section 5.3), so the finding that LLFC implies LMC (Lemma 1) is formalizing a well-established phenomenon which is somewhat obvious.”**\n\n**A1**: We'd like to emphasize that the interpretation that current alignment algorithms optimize for LLFC (Sec 5.3) is a new finding and one of the key contributions of this paper. This provides a deeper theoretical support for the alignment algorithms, which was absent in existing literature. See Reviewer Dt29's comments: \"The fact that both the weight-matching and activation-matching losses can be found in the commutativity condition is quite neat, as (to the best of our knowledge) these two methods were proposed as heuristics.\" We respectfully disagree that any of our findings were well-established, since the notion of LLFC never appeared in any prior work.\n\nWe'd also like to clarify the main contributions and logic progression of this paper:\n\n- We formulate LLFC, which is a stronger generalization of LMC (Lemma 1).\n- We empirically find that LMC networks always satisfy LLFC in practice, without exception (Sec 4).\n- We identify two conditions (weak additivity and commutativity) which imply LLFC, and verify them empirically.\n\nTogether, our work establishes LLFC as a more fine-grained and fundamental phenomenon than LMC. This novel perspective contributes to the new understanding of LMC and the spawning and permutation methods. Lemma 1 is only to formally verify that LLFC is stronger than LMC, but is not the main technical contribution of this paper.\n\n**Q2: “LLFC implying LMC is somehow less interesting than LMC implying LLFC, but the latter direction is not explored. Only co-occurence is observed experimentally, making the direction of causation unclear…”**\n\n**A2**: We have done a significant amount of experiments to verify \"LMC implies LLFC in practice\", i.e., all existing LMC networks satisfy LLFC, without exception (Sec 4). This shows that LLFC is a more fundamental phenomenon that may be the underlying cause of LMC in practice, and we make use of this perspective to obtain new insights (Sec 5). For the reverse direction, it is unlikely to establish LMC implies LLFC unconditionally, because LMC is a global property that only concerns the network output and LLFC characterizes the finer details in each layer in the network.\n\n**Q3: “For spawning LMC networks, it is unclear how the early training epochs contribute to enabling LMC. The experiments do not consider the evolution of LMC networks through training time.” and “how LLFC relates to the onset of LMC at an early point in training”**\n\n**A3**: Thank you for the comment. In Sec 5.2, we find that two spawned networks share similar principal directions in model weights, which contributes to the satisfaction of commutativity condition, thus leading to the satisfaction of LLFC and thereby LMC. This indicates that the early training epochs determine the top principal directions of the weights. In light of your suggestion, as shown in Figure 6 (global response), we conduct new experiments to evaluate the relationship between the number of shared early training epochs and the similarity between principal directions of spawned networks. We kindly refer you to the global response for more experimental details.\n\n**Q4: About baselines for comparison. “Figure 2 and 3: no cosine similarity for interpolated networks that are not linearly connected…”, “Figure 4: no sparsity baseline for random networks…” and “Figure 5: no commutativity distance for non-LMC networks…”**\n\n**A4**: Thank you for your suggestions. As shown in Figure 2 to 4 (global response), we conduct new experiments to add baselines for comparison. We kindly refer you to the global response for more experimental details.\n\n**Q5: “Section 5.2: how does similarity of $U_A$ and $U_B$ imply commutativity when $W=U\\Sigma V$ also depends on $V$, which could differ significantly between two networks?...”** \n\n**A5**: Thank you for your comment. As shown in Figure 5 (global response), we have verified the similarity between $V_A$ and $V_B$, and had the same observation as $U$. We kindly refer you to the global response for the experimental details. We will add this to the final version.\n\n**Q6: “How much would solving the quadratic assignment problem improve over existing algorithms? …it is not obvious whether minimizing the QAP will lead to significant improvements…”**\n\n**A6**: Solving QAPs is an NP-hard problem (Appendix C). Designing an efficient practical algorithms to QAPs is a challenging open problem on its own, which is beyond the scope of this paper. While we are currently unable to solve the QAP corresponding to Eq. (9), it is possible that solving the QAP will allow LMC to happen under weaker conditions (e.g. under a smaller network width compared to Git Re-Basin) because it enlarges the solution space compared with activation and weight matching.\n\n**Q7: “Weak additivity seems to rely heavily on the ReLU function. Does LLFC occur for other activation functions?”**\n\n**A7**: We did some preliminary experiments which showed that LMC and LLFC are difficult to achieve for other activation functions like tanh and sigmoid, so activation does play a role in LMC/LLFC. Since ReLU is the activation used in standard architectures including ResNet, VGG, etc. and is used in all previous work on LMC, an analysis based on ReLU is already highly informative given its wide adoption. We leave the exploration beyond the ReLU activation as a future direction.\n\n**Q8: “Do networks exist where LMC holds but not LLFC?…”**\n\n**A8**: No, we did not find any networks where LMC holds but not LLFC. We have conducted extensive experiments across diverse datasets, and model architectures (Sec 4 & Appendix D.2)."}, {"review_id": "S0IoiPVmBA", "reviewer": "Reviewer_Vyah", "summary": "There is a continuing effort to understand the complex training dynamics and loss landscape of neural networks, and one of the most interesting discoveries is Linear Mode Connectivity (LMC). LMC is the phenomenon that when two different solutions are linearly interpolated in parameter space, the training loss and test loss remain low enough to be similar to the solution. In this paper, the authors propose Layerwise Linear Feature Connectivity (LLFC), a stronger linear connectivity than LMC. LLFC means that the feature maps of all layers of two differently trained solutions are also linearly connected. Through various experiments, they show that LLFC is satisfied when spawning and permutation methods, which are common methods for presenting LMC characteristics, are used. In addition, they advance the understanding of LMC with a theoretical explanation of the underlying factors that make LLFC appear naturally.", "questions": "- I'm not sure how low the value of the sparsity measure is in Figure 4. Is a value of ~0.3 considered sparse?\n- In the experiments, the values measured with random variables, etc. are used as baseline, but it would be more accurate to compare them to the values for a model that is not linearly connected. The baseline values currently presented seem to exaggerate the results because they are comparing too different components.\n- Just a question, do you think the ensemble performance would be improved when ensembling linearly interpolated features with the modes?", "limitations": "I see no potential negative societal impact from this paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper introduced LLFC, an interesting phenomenon that extends LMC, linear connectivity in parameter space, to linear connectivity in feature space.\n- The empirical results are comprehensive and contain the necessary contents for the development of the discussion.\n- The theoretical analysis provides a convincing explanation for the emergence of LLFC.", "weaknesses": "- The authors only ran their experiments on MNIST and CIFAR10, which are relatively easy datasets. It will be important to verify its validity on larger datasets such as ImageNet. Git Re-Basin (Ainsworth at el., 2023) also showed a relatively large loss barrier on ImageNet, so validation on a wider range of datasets is required.\n-----\n(Ainsworth at el., 2023) [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://openreview.net/forum?id=CQsmMYmlP5T)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Ask for additional experiments on larger datasets such as ImageNet. “The authors only ran their experiments on MNIST and CIFAR10, which are *relatively easy* datasets. It will be important to verify its validity on larger datasets such as ImageNet.”**\n\n**A1**: Thank you for great suggestions. To begin with, we want to clarify that the purpose of our main experiments is to verify that LLFC co-occurs with LMC. Because the permutation methods cannot achieve the zero-loss-barrier LMC on ImageNet dataset [1], we can not verify that LLFC co-occurs with LMC in this particular instance. However, we take note that Frankle et al. [8] observed LMC on ResNet-50 trained on ImageNet dataset using the spawning method. Therefore, we follow your suggestion and conduct additional experiments on larger datasets using the spawning method. Given the temporal constraints of the Rebuttal period and the limitations of computational resources, we opted for the Tiny-ImageNet dataset [Cite 1], which is smaller but harder challenge than ImageNet. \n\nOur experiments adhere to the identical training configurations as those outlined in Frankle et al. [8]. We apply spawning method to obtain the two linearly connected modes, $\\boldsymbol{\\theta_A}$ and $\\boldsymbol{\\theta}_B$. Subsequently, consistent with the experimental settings of the main paper, we evaluate both ${\\rm cosine}\\_{\\alpha}(\\boldsymbol{x}_i)$ and ${\\rm cosine}\\_{A, B}(\\boldsymbol{x}_i)$ for each data point $\\boldsymbol{x}_i$ in the test set $\\mathcal{D}$. In Figure 1 (global response), the values of $\\mathbb{E}\\_{\\mathcal{D}}[1-{\\rm cosine}\\_{\\alpha}(\\boldsymbol{x}_i)]$ consistently approximate zero in contrast to $\\mathbb{E}\\_{\\mathcal{D}}[1-{\\rm cosine}\\_{A,B}(\\boldsymbol{x}_i)]$ across different layers and different values of $\\alpha$. This further strengthens our argument, confirming the co-occurrence of LLFC and LMC. For more experiment details, please kindly refer to our global response.\n\n[cite 1] Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015\n\n**Q2: Ask for a baseline of sparsity for comparison. “I'm not sure how low the value of the sparsity measure is in Figure 4.”**\n\n**A2**: Thank you for great question. We follow your suggestion and conduct new experiments to add baselines of sparsity for comparison. We choose the pre-activations of random initialized networks as our baseline. We measure the sparsity of the pre-activations of both well-trained networks and random initialized networks, using $S(\\boldsymbol{x}) = \\frac{\\|\\boldsymbol{x}\\|_1}{n\\|\\boldsymbol{x}\\|\\_{\\infty}}(\\boldsymbol{x} \\in \\mathbb{R}^n)$, denoted as $S(\\tilde{\\boldsymbol{h}}\\_{i, \\text{end}})$ and $S(\\tilde{\\boldsymbol{h}}\\_{i, \\text{init}})$ respectively. In Figure 3 (global response), the values of $S(\\tilde{\\boldsymbol{h}}\\_{i, \\text{end}})$ are relatively small compared to $S(\\tilde{\\boldsymbol{h}}\\_{i, \\text{init}})$, thus providing further supports for our sparsity claim. For more experimental details, please kindly refer to our global response.\n\n**Q3: Ask for more baseline for verifying weak additivity condition. “In the experiments, the values measured with random variables, etc. are used as baseline, but it would be more accurate to compare them to the values for a model that is not linearly connected.”**\n\n**A3**: Thank you for your great suggestion. We follow your suggestion and conduct additional experiments to comparing with models that are not linearly connected. Specifically, we compare $\\text{Dist}\\_{\\sigma}(\\tilde{\\boldsymbol{h}}\\_{i, A}, \\tilde{\\boldsymbol{h}}\\_{i, B})$ with $\\text{Dist}\\_{\\sigma}(\\tilde{\\boldsymbol{h}}\\_{i, C}, \\tilde{\\boldsymbol{h}}\\_{i, D})$ and $\\text{Dist}\\_{\\sigma}(\\boldsymbol{r}_1, \\boldsymbol{r}_2)$. Here, $\\tilde{\\boldsymbol{h}}\\_{i, A}, \\tilde{\\boldsymbol{h}}\\_{i, B}$ denote the pre-activations of two linearly connected mode $\\boldsymbol{\\theta}\\_A$ and $\\boldsymbol{\\theta}\\_B$, while $\\tilde{\\boldsymbol{h}}\\_{i, C}, \\tilde{\\boldsymbol{h}}\\_{i, D}$ denote the pre-activations of two independently trained mode $\\boldsymbol{\\theta}_C$ and $\\boldsymbol{\\theta}_D$.  Meanwhile, $\\boldsymbol{r}_1$ and $\\boldsymbol{r}_2$ are still independent $d\\_{\\ell}$-dimensional random vectors sampled from $\\mathcal{N}(\\boldsymbol{0}, \\boldsymbol{I})$. In Figure 3 (global response), the values of $\\text{Dist}\\_{\\sigma}(\\tilde{\\boldsymbol{h}}\\_{i, A}, \\tilde{\\boldsymbol{h}}\\_{i, B})$ are negligible in comparison to $\\text{Dist}\\_{\\sigma}(\\tilde{\\boldsymbol{h}}\\_{i, C}, \\tilde{\\boldsymbol{h}}\\_{i, D})$ and $\\text{Dist}\\_{\\sigma}(\\boldsymbol{r}_1, \\boldsymbol{r}_2)$, which further validate the weak additivity condition for linearly connected models. For more experiment details, please kindly refer to our global response.\n\n**Q4: Question on ensemble performance. “Just a question, do you think the ensemble performance would be improved when ensembling linearly interpolated features with the modes?”**\n\n**A4**: Thank you for the interesting question. While we do not have a definitive answer to this question, we do wish to highlight several studies that bear relevance. For example, as explored in Section 5.4 of Ainsworth et al. [1], an interesting observation arises when training two models on distinct splits of the dataset. In this context, the performance of a linearly interpolated model fails to match that of an ensemble of two models with twice the number of effective weights. This intriguing issue holds potential for further investigation and we can get insights to enhance the design of more effective ensemble methods."}]

Author rebuttal:
**Limitations.**

1. In Appendix C, identifying a permutation that directly enforces commutativity condition involves solving a NP-hard QAP. We leave the QAP-solving problem as a future direction.
2. Our Theorem 1 predicts LLFC in an ideal case, while in practice, a scaling factor $c$ is introduced to the definition of LLFC to better describe the experimental results. Realistic theorems and definitions (approximated version) are deferred to future research.
3. Our current experiments mainly focus on image classification, aligning with existing literature on LMC. While we appreciate suggestions to extend empirical evidence beyond image classification, we commit to exploring this avenue in future research.

**Experimental details.**

**[Figure 1] Further verify LLFC under various settings.**

To verify the LLFC property, we measure ${\rm cosine}\_{\alpha}(\boldsymbol{x}_i)$ and compare with ${\rm cosine}\_{A, B}(\boldsymbol{x}_i)$, consistently with main paper. In Fig 1, the values of $\mathbb{E}\_{\mathcal{D}}[1-{\rm cosine}\_{\alpha}(\boldsymbol{x}_i)]$ are close to 0 compared with $\mathbb{E}\_{\mathcal{D}}[1-\text{cosine}\_{A, B}(\boldsymbol{x}_i)]$, and thus convincingly verify our claim. 

Notably, all the experimental settings are standard: the settings of MLPs on the MNIST and CIFAR-10 follows Ainsworth et al. 2022 [1]; the training of ResNet-20 on the CIFAR-10 follows the default settings of Pytorch; the training of ResNet-50 on the Tiny ImageNet follows Frankle et al. [8].  

**[Figure 2] Add baseline of non-LMC models for verifying LLFC.**

We measure ${\rm cosine}_{0.5}(\boldsymbol{x}_i)$ on both the linearly connected models and independently trained models, denoted as ${\rm cosine}\_{LMC}(\boldsymbol{x}_i)$ and ${\rm cosine}\_{not \ LMC}(\boldsymbol{x}_i)$ correspondingly. In Fig 2, the values of $\mathbb{E}\_{\mathcal{D}}[1-{\rm cosine}\_{LMC}(\boldsymbol{x}_i)]$ are negligible compared to $\mathbb{E}\_{\mathcal{D}}[1-{\rm cosine}\_{not \ LMC}(\boldsymbol{x}_i)]$. Therefore, we rule out the possibility that LLFC is a byproduct of averaging.

**[Figure 3] Add baselines for verifying weak additivity and sparsity.**

First, to verify the weak additivity , we compared ${\rm Dist}\_{\sigma}(\tilde{\boldsymbol{h}}\_{i, A}, \tilde{\boldsymbol{h}}\_{i, B})$ with ${\rm Dist}\_{\sigma}(\tilde{\boldsymbol{h}}\_{i, C}, \tilde{\boldsymbol{h}}\_{i, D})$ and ${\rm Dist}\_{\sigma}(\boldsymbol{r}_1, \boldsymbol{r}_2)$. The subscripts $A, B, C, D$ denote four different models, where $A, B$ are linearly connected and $C, D$ are independently trained. In Fig 3, the values of ${\rm Dist}\_{\sigma}(\tilde{\boldsymbol{h}}\_{i, A}, \tilde{\boldsymbol{h}}\_{i, B})$ are negligible compared with ${\rm Dist}\_{\sigma}(\tilde{\boldsymbol{h}}\_{i, C}, \tilde{\boldsymbol{h}}\_{i, D})$ and ${\rm Dist}\_{\sigma}(\boldsymbol{r}_1, \boldsymbol{r}_2)$. Therefore, we verify that the weak additivity condition holds for modes that are linearly connected.

Second, to verify the sparsity claim, we measure the sparsity of the pre-activations of both well-trained networks and random initialized networks, using $S(\boldsymbol{x}) = \frac{\|\boldsymbol{x}\|_1}{n\|\boldsymbol{x}\|\_{\infty}}(\boldsymbol{x} \in \mathbb{R}^n)$, denoted as $S(\tilde{\boldsymbol{h}}\_{i, \text{end}})$ and $S(\tilde{\boldsymbol{h}}\_{i, \text{init}})$ respectively. In Fig 3, the majority of the values of $S(\tilde{\boldsymbol{h}}\_{i, \text{end}})$ are distinctively smaller than $S(\tilde{\boldsymbol{h}}\_{i, \text{init}})$. Therefore, we validate our sparsity claim.

Notably, in Fig 3, the values of ${\rm Dist}\_{\sigma}(\tilde{\boldsymbol{h}}\_{i, C}, \tilde{\boldsymbol{h}}\_{i, D})$ are smaller than ${\rm Dist}\_{\sigma}(\boldsymbol{r}_1, \boldsymbol{r}_2)$. The gap between between ${\rm Dist}\_{\sigma}(\tilde{\boldsymbol{h}}\_{i, C}, \tilde{\boldsymbol{h}}\_{i, D})$ and ${\rm Dist}\_{\sigma}(\boldsymbol{r}_1, \boldsymbol{r}_2)$ increases as the gap between the gap between $S(\tilde{\boldsymbol{h}}\_{i, \text{end}})$ and $S(\tilde{\boldsymbol{h}}\_{i, \text{init}})$ widens. This observation supports that the weak additivity is likely to hold if the pre-activations are sparse enough.

**[Figure 4] Add baselines of non-LMC models for verifying commutativity.**

We measure ${\rm Dist}\_{com}$ on both the linearly connected models and independently trained models, denoted as ${\rm Dist}\_{com, LMC}$ and ${\rm Dist}\_{com, not\ LMC}$. In Fig 4, the values of ${\rm Dist}\_{com, LMC}$ are negligible compared with ${\rm Dist}\_{com, not\ LMC}$, which validates the commutativity condition for LMC models.

**[Figure 5] Verify the top singular vectors of $V$ hold a small principal angle similar to $U$.**

We extract the top $k$ singular vectors from $V_A$ and $V_B$ and compute the minimal principal angle $\beta_{V}$ between the subspaces spanned by these singular vectors. We compare the principal angle of two linearly connected models, $\beta_{LMC, V}$, with that of two independently trained models, $\beta_{not \ LMC, V}$. In Fig 5, the angle $1-\cos\beta_{LMC, V}$ is close to 0 while $1-\cos\beta_{not\ LMC, V}$ is significantly large, thus confirming our claim.

**[Figure 6] Relationship between shared early training epochs and the similarity between principal directions of spawned models.**

We compute both $1-\cos\beta_{U}$ and $1-\cos\beta_{V}$ for two models that spawning at different iteration $t$.  In Fig 6, across different layers, both $1-\cos\beta_{U}$ and $1-\cos\beta_{V}$ decrease as the spawning iteration increases. This implies that similarity between principal directions of spawned models increases if sharing more early training iterations.

Furthermore, a noticeable similarity emerges between the curves of $1-\cos\beta$ v.s. spawning iteration $t$ and the curves of the instability (defined in Frankle et al. [8]) v.s. the spawning iteration $t$ (Fig 3 in Frankle et al. [8]). This enhances the credibility of our analysis significantly.

Meta-review: This paper considers the phenomenon of linear model connectivity in trained neural networks. The main contribution is to provide empirical evidence of an even stronger notion of linear connectivity: Layerwise Linear Feature Connectivity (LLFC). This means that the feature maps of every layer can be connected.

The reviewers agree that this is an interesting contribution and the very detailed response of the authors has helped resolve most of the issues raised in the initial reviews. After the rebuttal, there is a clear consensus towards accepting the paper. After my own reading of the reviews, rebuttal and paper, I agree with this view and recommend acceptance. I warmly encourage the authors to incorporate the additional numerical results, experimental details and discussions from the rebuttal phase in the final version.
Decision: Accept (poster)