Title: Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivity
Venue: NeurIPS 2023
Accepted: yes
Keywords: Linear Mode Connectivity

Permutation Invariance

Optimization Landscape

Science of Deep Learning
Abstract: Recent work has revealed many intriguing empirical phenomena in neural network training, despite the poorly understood and highly complex loss landscapes and training dynamics. One of these phenomena, Linear Mode Connectivity (LMC), has gained considerable attention due to the intriguing observation that different solutions can be connected by a linear path in the parameter space while maintaining near-constant training and test losses. In this work, we introduce a stronger notion of linear connectivity, Layerwise Linear Feature Connectivity (LLFC), which says that the feature maps of every layer in different trained networks are also linearly connected. We provide comprehensive empirical evidence for LLFC across a wide range of settings, demonstrating that whenever two trained networks satisfy LMC (via either spawning or permutation methods), they also satisfy LLFC in nearly all the layers. Furthermore, we delve deeper into the underlying factors contributing to LLFC, which reveal new insights into the permutation approaches. The study of LLFC transcends and advances our understanding of LMC by adopting a feature-learning perspective.

1

Abstract

Recent work has revealed many intriguing empirical phenomena in neural network training, despite the poorly understood and highly complex loss landscapes and training dynamics. One of these phenomena, Linear Mode Connectivity (LMC), has gained considerable attention due to the intriguing observation that different solutions can be connected by a linear path in the parameter space while maintaining near-constant training and test losses. In this work, we introduce a stronger notion of linear connectivity, Layerwise Linear Feature Connectivity (LLFC) , which says that the feature maps of every layer in different trained networks are also linearly connected. We provide comprehensive empirical evidence for LLFC across a wide range of settings, demonstrating that whenever two trained networks satisfy LMC (via either spawning or permutation methods), they also satisfy LLFC in nearly all the layers. Furthermore, we delve deeper into the underlying factors contributing to LLFC, which reveal new insights into the permutation approaches. The study of LLFC transcends and advances our understanding of LMC by adopting a feature-learning perspective. We released our source code at https://github.com/zzp1012/LLFC .

1 Introduction

Despite the successes of modern deep neural networks, theoretical understanding of them still lags behind. Efforts to understand the mechanisms behind deep learning have led to significant interest in exploring the loss landscapes and training dynamics. While the loss functions used in deep learning are often regarded as complex black-box functions in high dimensions, it is believed that these functions, particularly the parts encountered in practical training trajectories, contain intricate benign structures that play a role in facilitating the effectiveness of gradient-based training [36, 4, 39]. Just like in many other scientific disciplines, a crucial step toward formulating a comprehensive theory of deep learning lies in meticulous empirical investigations of the learning pipeline, intending to uncover quantitative and reproducible nontrivial phenomena that shed light on the underlying mechanisms.

One intriguing phenomenon discovered in recent work is Mode Connectivity [10, 5, 11]: Different optima found by independent runs of gradient-based optimization are connected by a simple path in the parameter space, on which the loss or accuracy is nearly constant. This is surprising as different optima of a non-convex function can very well reside in different and isolated 'valleys' and yet this does not happen for optima found in practice. More recently, an even stronger form of mode

Overview

Version 3

∝ 𝛼

+(1 - 𝛼)

Feature maps of model 𝜃𝛼 are proportional to linear interpolation of feature maps of model 𝜃𝐴 and 𝜃𝐵 , where 𝜃𝛼 = 𝛼𝜃𝐴 + 1 - 𝛼 𝜃𝐵 .

𝛼 = 1

𝛼 = 0

𝛼 =

3

4

𝛼 =

1

2

𝛼 =

1

4

𝑓 (ℓ)

𝜃𝛼;

=

Feature maps of

𝜃𝐵

Feature maps of

𝜃𝐴

Feature maps of interpolated model

𝜃𝛼

connectivity called Linear Mode Connectivity (LMC) was discovered [24, 9]. It depicts that different optima can be connected by a linear path of constant loss/accuracy. Although LMC typically does not happen for two independently trained networks, it has been consistently observed in the following scenarios:

Spawning [9, 7]: A network is randomly initialized, trained for a small number of epochs (e.g. 5 epochs for both ResNet-20 and VGG-16 on the CIFAR-10 dataset), and then spawned into two copies which continue to be independently trained using different SGD randomnesses (i.e., for mini-batch order and data augmentation).

Permutation [6, 1]: Two networks are independently trained, and the neurons of one model are permuted to match the neurons of the other model while maintaining a functionally equivalent network.

The study of LMC is highly motivated due to its ability to unveil nontrivial structural properties of loss landscapes and training dynamics. Furthermore, LMC has significant relevance to various practical applications, such as pruning and weight-averaging methods.

On the other hand, the success of deep neural networks is related to their ability to learn useful features , or representations , of the data [27], and recent work has highlighted the importance of analyzing not only the final outputs of a network but also its intermediate features [20]. However, this crucial perspective is absent in the existing literature on LMC and weight averaging. These studies typically focus on interpolating the weights of different models and examining the final loss and accuracy, without delving into the internal layers of the network.

In this work, we take a feature-learning perspective on LMC and pose the question: what happens to the internal features when we linearly interpolate the weights of two trained networks? Our main discovery, referred to as Layerwise Linear Feature Connectivity (LLFC) , is that the features in almost all the layers also satisfy a strong form of linear connectivity: the feature map in the weight-interpolated network is approximately the same as the linear interpolation of the feature maps in the two original networks. More precisely, let θ A and θ B be the weights of two trained networks, and let f ( ℓ ) ( θ ) be the feature map in the ℓ -th layer of the network with weights θ . Then we say that θ A and θ B satisfy LLFC if

See Figure 1 for an illustration. While LLFC certainly cannot hold for two arbitrary θ A and θ B , we find that it is satisfied whenever θ A and θ B satisfy LMC. We confirm this across a wide range of settings, as well as for both spawning and permutation methods that give rise to LMC.

LLFC is a much finer-grained characterization of linearity than LMC. While LMC only concerns loss or accuracy, which is a single scalar value, LLFC (1) establishes a relation for all intermediate feature maps, which are high-dimensional objects. Furthermore, it is not difficult to see that LLFC applied to the output layer implies LMC when the two networks have small errors (see Lemma 1); hence, LLFC can be viewed as a strictly stronger property than LMC. The consistent co-occurrence of LLFC and LMC suggests that studying LLFC may play a crucial role in enhancing our understanding of LMC.

Subsequently, we delve deeper into the underlying factors contributing to LLFC. We identify two critical conditions, weak additivity for ReLU function and a commutativity property between two trained networks. We prove that these two conditions collectively imply LLFC in ReLU networks, and provide empirical verification of these conditions. Furthermore, our investigation yields novel insights into the permutation approaches: we interpret both the activation matching and weight matching objectives in Git Re-Basin [1] as ways to ensure the satisfaction of commutativity property.

In summary, our work unveils a richer set of phenomena that go significantly beyond the scope of LMC, and our further investigation provides valuable insights into the underlying mechanism behind LMC. Our results demonstrate the value of opening the black box of neural networks and taking a feature-centric viewpoint in studying questions related to loss landscapes and training dynamics.

2 Related Work

(Linear) Mode Connectivity. Freeman and Bruna [10], Draxler et al. [5], Garipov et al. [11] observed Mode Connectivity, i.e., different optima/modes of the loss function can be connected through a non-linear path with nearly constant loss. Nagarajan and Kolter [24] first observed Linear Mode Connectivity (LMC), i.e., the near-constant-loss connecting path can be linear, on models trained on MNIST starting from the same random initialization. Later, Frankle et al. [9] first formally defined and thoroughly investigate the LMC problem. Frankle et al. [9] observed LMC on harder datasets, for networks that are jointly trained for a short amount of time before going through independent training (we refer to this as the spawning method). Frankle et al. [9] also demonstrated a connection between LMC and the Lottery Ticket Hypothesis [8]. Fort et al. [7] used the spawning method to explore the connection between LMC and the Neural Tangent Kernel dynamics. Lubana et al. [23] studied the mechanisms of DNNs from mode connectivity and verified the mechanistically dissimilar models cannot be linearly connected. Yunis et al. [38] showed that LMC also extends beyond two optima and identified a high-dimensional convex hull of low loss between multiple optima. On the theory side, several papers [10, 21, 32, 26, 25, 16] were able to prove non-linear mode connectivity under various settings, but there has not been a theoretical explanation of LMC to our knowledge.

Permutation Invariance. Neural network architectures contain permutation symmetries [13]: one can permute the weights in different layers while not changing the function computed by the network. Ashmore and Gashler [2] utilized the permutation invariance of DNNs to align the topological structure of two neural networks. Tatro et al. [31] used permutation invariance to align the neurons of two neural networks, resulting in improved non-linear mode connectivity. In the context of LMC, Entezari et al. [6], Ainsworth et al. [1] showed that even independently trained networks can be linearly connected when permutation invariance is taken into account. In particular, Ainsworth et al. [1] approached the neuron alignment problem by formulating it as bipartite graph matching and proposed two matching methods: activation matching and weight matching. Notably, Ainsworth et al. [1] achieved the LMC between independently trained ResNet models on the CIFAR-10 dataset using weight matching.

Model Averaging Methods. LMC also has direct implications for model averaging methods, which are further related to federated learning and ensemble methods. Wang et al. [33] introduced a novel federated learning algorithm that incorporates unit permutation before model averaging. Singh and Jaggi [30], Liu et al. [22] approached the neuron alignment problem in model averaging by formulating it as an optimal transport and graph matching problem, respectively. Wortsman et al. [35] averaged the weights of multiple fine-tuned models trained with different hyper-parameters and obtained improved performance.

3 Background and Preliminaries

Notation and Setup. Denote [ k ] = { 1 , 2 , . . . , k } . We consider a classification dataset D = { ( x i , y i ) } n i =1 , where x i ∈ R d 0 represents the input and y i ∈ [ c ] represents the label of the i -th data point. Here, n is the dataset size, d 0 is the input dimension and c is the number of classes. Moreover, we use X ∈ R d 0 × n to stack all the input data into a matrix.

We consider an L -layer neural network of the form f ( θ ; x ) , where θ represents the model parameters, x is the input, and f ( θ ; x ) ∈ R c . Let the ℓ -th layer feature (post-activation) of the network be f ( ℓ ) ( θ ; x ) ∈ R d ℓ , where d ℓ is the dimension of the ℓ -th layer ( 0 ≤ ℓ ≤ L ) and d L = c . Note that

f (0) ( θ ; x ) = x and f ( L ) ( θ ; x ) = f ( θ ; x ) . For an input data matrix X , we also use f ( θ ; X ) ∈ R c × n and f ( ℓ ) ( θ ; X ) ∈ R d ℓ × n to denote the collection of the network outputs and features on all the datapoints, respectively. When X is clear from the context, we simply write f ( ℓ ) ( θ ) = f ( ℓ ) ( θ ; X ) and f ( θ ) = f ( θ ; X ) . Unless otherwise specified, in this paper we consider models trained on a training set, and then all the investigations are evaluated on a test set.

We use Err D ( θ ) to denote the classification error of the network f ( θ ; · ) on the dataset D .

Linear Mode Connectivity (LMC). We recall the notion of LMC in Definition 1.

Definition 1 ( Linear Mode Connectivity ) . Given a test dataset D and two modes 3 θ A and θ B such that Err D ( θ A ) ≈ Err D ( θ B ) , we say θ A and θ B are linearly connected if they satisfy

As Definition 1 shows, θ A and θ B satisfy LMC if the error metric on the linear path connecting their weights is nearly constant. There are two known methods to obtain linearly connected modes, the spawning method [9, 7] and the permutation method [6, 1].

Spawning Method. We start from random initialization θ (0) and train the model for k steps to obtain θ ( k ) . Then we create two copies of θ ( k ) and continue training the two models separately using independent SGD randomnesses (mini-batch order and data augmentations) until convergence. By selecting a proper value of k (usually a small fraction of the total training steps), we can obtain two linearly connected modes.

Permutation Method. Due to permutation symmetry, it is possible to permute the weights in a neural network appropriately while not changing the function being computed. Given two modes θ A and θ B which are independently trained (and not linearly connected), the permutation method aims to find a permutation π such that the permuted mode θ ′ B = π ( θ B ) is functionally equivalent to θ B and that θ ′ B and θ A are linearly connected. In other words, even if two modes are not linearly connected in the parameter space, they might still be linearly connected if permutation invariance is taken into account.

Among existing permutation methods, Git Re-Basin [1] is a representative one, which successfully achieved linear connectivity between two independently trained ResNet models on CIFAR-10. Specifically, a permutation π that maintains functionally equivalent network can be formulated by a set of per-layer permutations π = { P ( ℓ ) } L -1 ℓ =1 where P ( ℓ ) ∈ R d ℓ × d ℓ is a permutation matrix. Ainsworth et al. [1] proposed two distinct matching objectives for aligning the neurons of independently trained models via permutation: weight matching and activation matching :

Here, H ( ℓ ) A = f ( ℓ ) ( θ A ; X ) is the ℓ -th layer feature matrix, and H ( ℓ ) B is defined similarly.

Main Experimental Setup. In this paper, we use both the spawning method and the permutation method to obtain linearly connected modes. Following Frankle et al. [9], Ainsworth et al. [1], we perform our experiments on commonly used image classification datasets MNIST [18], CIFAR10 [15], and Tiny-ImageNet[17], and with the standard network architectures ResNet-20/50 [12], VGG-16 [29], and MLP. We follow the same training procedures and hyper-parameters as in Frankle et al. [9], Ainsworth et al. [1]. Due to space limit, we defer some of the experimental results to the appendix. Notice that [1] increased the width of ResNet by 32 times in order to achieve zero barrier, and we also followed this setting in the experiments of the permutation method. The detailed settings and hyper-parameters are also described in Appendix B.1.

4 Layerwise Linear Feature Connectivity (LLFC)

In this section, we formally describe Layerwise Linear Feature Connectivity (LLFC) and provide empirical evidence of its consistent co-occurrence with LMC. We also show that LLFC applied to the last layer directly implies LMC.

Verification of Additivity

Version 5

Verification of Additivity

Version 5

Dist 𝛼

= 1 - cos⟨𝑓 𝑙

Dist 𝐴.𝐵

1 - 𝛼 𝜃𝐵

= 1 - cos⟨𝑓 𝑙

𝑓 𝑙

∈ 𝑅 𝑛×𝑑

MLP on MNIST

FC 1

FC 2

FC 3

FC 4

ResNet20 on CIFAR10

Block 1-2

Block 1-3

Block 2-2

Block 2-3

Block 3-2

Block 3-3

FC

Block 2-4

Block 3-1

Block 3-2

Block 3-3

Block 3-4

Block 3-5

Block 3-6

Block 4-1

Block 4-2

Block 4-3

FC

1 - cosine0.25

1 - cosine0.50

1 - cosine0.75

1 - cosine 𝐴,𝐵

𝐸𝐷[1 - cosine 𝒙𝑖

]

𝐸𝐷[1 - cosine 𝒙𝑖

]

Conv 4-2

Conv 4-3

Conv 5-2

Conv 5-3

FC 1

FC 2

FC 3

Conv 1-2

Conv 2-2

Conv 3-2

Conv 3-3

𝐸𝐷[1 - cosine 𝒙𝑖

]

VGG16 on CIFAR10

ResNet50 on Tiny ImageNet

Dist 𝛼

= 1 - cos⟨𝑓 𝑙

1 - 𝛼 𝜃𝐵

, 𝛼𝑓

𝑙

𝜃𝐴

+ 1 - 𝛼 𝑓 𝑙 (𝜃 𝐵

Dist 𝐴.𝐵

= 1 - cos⟨𝑓 𝑙

𝜃𝐴

, 𝑓

𝑙

(𝜃𝐵 )⟩

𝑓 𝑙

∈ 𝑅 𝑛×𝑑

0.5

0.0

0.5

0.0

0.5

0.0

工

工

工

工工工工

ResNet20 (32 x) on CIFAR-10 (Weight Matching)

Block 1-1

Block 1-2

Block 1-3

Block 2-1

Block 2-2

Block 3-1

Block 3-2

Block 3-3

FC

Block 2-3

1 - cosine0.25

1 - cosine0.50

1 - cosine0.75

1 - cosine 𝐴,𝐵

𝐸𝐷[1 - cosine 𝒙𝑖

]

MLP on MNIST (Activation Matching)

FC 1

FC 2

FC 3

FC 4

MLP on MNIST (weight matching)

FC 1

FC 2

FC 3

FC 4

𝐸𝐷[1 - cosine 𝒙𝑖

]

𝐸𝐷[1 - cosine 𝒙𝑖

]

MLP on CIFAR-10 (Activation Matching)

FC 1

FC 2

FC 3

FC 4

MLP on CIFAR-10 (weight matching)

FC 1

FC 2

FC 3

FC 4

𝐸𝐷[1 - cosine 𝒙𝑖

]

𝐸𝐷[1 - cosine 𝒙𝑖

]

0.5

工

工

0.0

0.5

工

0.0

0.5

0.0

工

工

0.5

工

0.0

工

0.5

0.0

Definition 2 ( Layerwise Linear Feature Connectivity ) . Given dataset D and two modes θ A , θ B of an L -layer neural network f , the modes θ A and θ B are said to be layerwise linearly feature connected if they satisfy

LLFC states that the per-layer feature of the interpolated model θ α = α θ A +(1 -α ) θ B has the same direction as the linear interpolation of the features of θ A and θ B . This means that the feature map f ( ℓ ) behaves similarly to a linear map (up to a scaling factor) on the line segment between θ A and θ B , even though it is a nonlinear map globally.

LLFC Co-occurs with LMC. We now verify that LLFC consistently co-occurs with LMC across different architectures and datasets. We use the spawning method and the permutation method described in Section 3 to obtain linearly connected modes θ A and θ B . On each data point x i in the test set D , we measure the cosine similarity between the feature of the interpolated model θ α and

, 𝛼𝑓

𝑙

𝜃𝐴

, 𝑓

𝑙

𝜃𝐴

+ 1 - 𝛼 𝑓 𝑙 (𝜃 𝐵

(𝜃𝐵 )⟩

Version 5

Dist 𝛼

= 1 - cos⟨𝑓 𝑙

Dist 𝐴.𝐵

1 - 𝛼 𝜃𝐵

= 1 - cos⟨𝑓 𝑙

𝑓 𝑙

∈ 𝑅 𝑛×𝑑

ResNet20 (32 x) on CIFAR10 (Weight Matching)

Block 2-3

Block 3-1

Block 3-2

Block 3-3

FC

Block 3-1

Block 3-2

Block 3-3

FC

Block 2-3

1 - cosine𝐿𝑀𝐶

1 - cosine𝑛𝑜𝑡 𝐿𝑀𝐶

𝐸𝐷[1 - cosine 𝒙𝑖

]

ResNet20 on CIFAR10 (Spawning)

linear interpolations of the features of θ A and θ B in each layer ℓ , as expressed as cosine α ( x i ) = cos[ f ( ℓ ) ( α θ A + (1 -α ) θ B ; x i ) , αf ( ℓ ) ( θ A ; x i ) + (1 -α ) f ( ℓ ) ( θ B ; x i )] . We compare this to the baseline cosine similarity between the features of θ A and θ B in the corresponding layer, namely cosine A,B ( x i ) = cos[ f ( ℓ ) ( θ A ; x i ) , f ( ℓ ) ( θ B ; x i )] . The results for the spawning method and the permutation method are presented in Figures 2 and 3, respectively. They show that the values of E D [1 -cosine α ( x i )] are close to 0 across different layers, architectures, datasets, and different values of α , which verifies the LLFC property. The presence of small error bars indicates consistent behavior for each data point. Moreover, the values of E D [1 -cosine A,B ( x i )] are not close to 0 , which rules out the trivial case that f ( ℓ ) ( θ A ) and f ( ℓ ) ( θ B ) are already perfectly aligned.

To further verify, we also compare the values of E D [1 -cosine α ( x i )] of two linearly connected modes with those of two modes that are independently trained (not satisfying LMC). We measure cosine 0 . 5 of the features of two modes that are linearly connected and two modes that are independently trained, denoted as cosine LMC and cosine not LMC correspondingly. In Figure 4, the values of E D [1 -cosine LMC ( x i )] are negligible compared to E D [1 -cosine not LMC ( x i )] . The experimental results align with Figures 2 and 3 and thus we firmly verify the claim that LLFC co-occurs with LMC.

LLFC Implies LMC. Intuitively, LLFC is a much stronger characterization than LMC since it establishes a linearity property in the high-dimensional feature map in every layer, rather than just for the final error. Lemma 1 below formally establishes that LMC is a consequence of LLFC by applying LLFC on the output layer.

Lemma 1 (Proof in Appendix A.1) . Suppose two modes θ A , θ B satisfy LLFC on a dataset D and max { Err D ( θ A ) , Err D ( θ B ) } ≤ ϵ , then we have

In summary, we see that the LMC property, which was used to study the loss landscapes in the entire parameter space, extends to the internal features in almost all the layers. LLFC offers much richer structural properties than LMC. In Section 5, we will dig deeper into the contributing factors to LLFC and leverage the insights to gain new understanding of the spawning and the permutation methods.

5 Why Does LLFC Emerge?

We have seen that LLFC is a prevalent phenomenon that co-occurs with LMC, and it establishes a broader notion of linear connectivity than LMC. In this section, we investigate the root cause of LLFC, and identify two key conditions, weak additivity for ReLU activations and commutativity . We verify these conditions empirically and prove that they collectively imply LLFC. From there, we provide an explanation for the effectiveness of the permutation method, offering new insights into LMC.

5.1 Underlying Factors of LLFC

For convenience, we consider a multi-layer perceptron (MLP) in Section 5, though the results can be easily adapted to any feed-forward structure, e.g., a convolutional neural network 4 (CNN). For

, 𝛼𝑓

𝑙

𝜃𝐴

, 𝑓

𝑙

𝜃𝐴

+ 1 - 𝛼 𝑓 𝑙 (𝜃 𝐵

(𝜃𝐵 )⟩

MLP on MNIST

FC 1

Distance

Dist 𝜎

Conv 5-3

Distance

Dist 𝜎

FC 2

Distance

Dist 𝜎

FC 3

Distance

Dist 𝜎

Conv 4-1

Distance

Dist 𝜎

Conv 4-2

Distance

Dist 𝜎

Conv 4-3

Distance

Dist 𝜎

Conv 5-1

Distance

Dist 𝜎

Conv 5-2

Distance

Dist 𝜎

VGG16 on CIFAR10

Dist 𝜎

( ෩

𝒉𝑖,𝐴

, ෩

𝒉𝑖,𝐵

)

Dist 𝜎

( ෩

𝒉𝑖,𝐶

, ෩

𝒉𝑖,𝐷 )

Dist 𝜎 =

||𝜎 ෩

𝒉1 + ෩

𝒉2

-𝒉1 - 𝒉2|| 2

||𝜎 ෩

𝒉1 + ෩

𝒉2

|| ||𝒉1 +𝒉2||

||𝜎

෤ 𝒓1

+ ෤ 𝒓 2

-𝒓1 - 𝒓2|| 2

FC 1

Distance

Dist 𝜎

FC 2

Distance

Dist 𝜎

FC 3

Distance

Dist 𝜎

MLP on MNIST

Block 1-1

Distance

Dist 𝜎

Block 1-2

Distance

Dist 𝜎

Block 2-1

Distance

Dist 𝜎

Block 2-2

Distance

Dist 𝜎

Block 3-1

Distance

Dist 𝜎

Block 3-2

Distance

Dist 𝜎

ResNet20 (32 x) on CIFAR10

Dist 𝜎

( ෩

𝒉𝑖,𝐶

, ෩

𝒉𝑖,𝐷𝑎𝑐𝑡

)

Dist 𝜎

( ෩

𝒉𝑖,𝐶

, ෩

𝒉𝑖,𝐷𝑤𝑔𝑡

)

Dist 𝜎

( ෩

𝒉𝑖,𝐶

, ෩

𝒉𝑖,𝐷 )

||𝜎

Dist 𝜎 =

෤ 𝒓1

+ ෤ 𝒓 2

|| ||𝒓1 +𝒓2||

Dist 𝜎 =

||𝜎 ෩

𝒉1 + ෩

||𝜎 ෩

||𝜎

𝒉2

𝒉1 + ෩

෤ 𝒓1

𝒉2

+ ෤ 𝒓 2

-𝒉1 - 𝒉2|| 2

|| ||𝒉1 +𝒉2||

-𝒓1 - 𝒓2|| 2

Dist 𝜎 = ||𝜎 ෤ 𝒓1 + ෤ 𝒓 2 || ||𝒓1 +𝒓2|| an L -layer MLP f with ReLU activation, the weight matrix in the ℓ -th linear layer is denoted as W ( ℓ ) ∈ R d ℓ × d ℓ -1 , and b ( ℓ ) ∈ R d ℓ is the bias in that layer. For a given input data matrix X ∈ R d 0 × n , denote the feature (post-activation) in the ℓ -th layer as H ( ℓ ) = f ( ℓ ) ( θ ; X ) ∈ R d ℓ × n , and correspondingly pre-activation as ˜ H ( ℓ ) ∈ R d ℓ × n . The forward propagation in the ℓ -th layer is:

Here, σ denotes the ReLU activation function, and 1 d ℓ ∈ R d ℓ denotes the all-one vector. Additionally, we use h ( ℓ ) i to denote the i -th row of H ( ℓ ) , and ˜ h ( ℓ ) i to denote the i -th row of ˜ H ( ℓ ) , which correspond to the post- and pre-activations of the i -th input at layer ℓ , respectively.

Condition I: Weak Additivity for ReLU Activations. 5

Definition 3 ( Weak Additivity for ReLU Activations ) . Given a dataset D , two modes θ A and θ B are said to satisfy weak additivity for ReLU activations if

Definition 3 requires the ReLU activation function to behave like a linear function for the preactivations in each layer of the two networks. Although this cannot be true in general since ReLU is a nonlinear function, we verify it empirically for modes that satisfy LMC and LLFC.

We conduct experiments on various datasets and architectures to validate the weak additivity for ReLU activations. Specifically, given two modes θ A and θ B and a data point x i in the test set D , we compute the normalized distances between the left-hand side and the right-hand side of Equation (7) for each layer ℓ , varying the values of α . We denote the maximum distance across the range of α as Dist σ ( ˜ h i,A , ˜ h i,B ) = max α ∈ [0 , 1] dist ( σ ( α ˜ h i,A +(1 -α ) ˜ h i,B ) , ασ ( ˜ h i,A ) + (1 -α ) σ ( ˜ h i,B )) where dist ( x , y ) := ∥ x -y ∥ 2 / ( ∥ x ∥ · ∥ y ∥ ) . To validate the weak additivity condition, we compare the values of Dist σ of two modes that are linearly connected with those of two modes that are independently trained.

Verification of Commutativity

Version 2

MLP on MNIST

ResNet20 on CIFAR10

FC

FC 2

FC 3

FC 4

Block 1-2

Block 1-3

Block 2-2

Block 2-3

Block 3-2

Block 3-3

FC

FC 2

FC 3

FC 4

Block 1-2

Block 1-3

Block 2-2

Block 2-3

Block 3-2

Block 3-3

MLP on MNIST (Weight Matching)

ResNet20 (32 x) on CIFAR10 (Weight Matching)

Dist 𝑐𝑜𝑚

Dist 𝑊

Dist 𝐻

Distance

Distance

Distance

Distance

Distance

Distance

Distance

Distance

Distance

Distance

Distance

Distance

Distance

Distance

Distance

Distance

Distance

Distance

Distance

Distance

Both spawning and permutation methods are shown to demonstrate the weak additivity condition. For spawning method, we first obtain two linearly connected modes, i.e, θ A and θ B , and then two independently trained modes, i.e, θ C and θ D (not satisfying LMC/LLFC). We compare the values of Dist σ of θ A and θ B with those of θ C and θ D . In Figure 5, we observe that across different datasets and model architectures, at different layers, Dist σ ( ˜ h i,A , ˜ h i,B ) are negligible (and much smaller than the baseline Dist σ ( ˜ h i,C , ˜ h i,D ) ). For permutation methods, given two independently trained modes, i.e., θ C and θ D , we permute the θ D such that the permuted π ( θ D ) are linearly connected with θ C . Both activation matching and weight matching are used and the permuted π ( θ D ) are denoted as θ D act and θ D wgt respectively. In Figure 6, the values of Dist σ ( ˜ h i,C , ˜ h i,D act ) and Dist σ ( ˜ h i,C , ˜ h i,D wgt ) are close to zero compared to Dist σ ( ˜ h i,C , ˜ h i,D ) . Therefore, we verify the weak additivity condition for both spawning and permutation methods.

Condition II: Commutativity.

Definition 4 ( Commutativity ) . Given a dataset D , two modes θ A and θ B are said to satisfy commutativity if

Commutativity depicts that the next-layer linear transformations applied to the internal features of two neural networks can be interchanged. This property is crucial for improving our understanding of LMC and LLFC. In Section 5.2, we will use the commutativity property to provide new insights into the permutation method.

We conduct experiments on various datasets and model architectures to verify the commutativity property for modes that satisfy LLFC. Specifically, for a given dataset D and two modes θ A and θ B that satisfy LLFC, we compute the normalized distance between the left-hand side and the right-hand side of Equation (8), denoted as Dist com = dist ( vec ( W ( ℓ ) A H ( ℓ -1) A + W ( ℓ ) B H ( ℓ -1) B ) , vec ( W ( ℓ ) A H ( ℓ -1) B + W ( ℓ ) B H ( ℓ -1) A ) ) . Furthermore, we compare Dist com with the normalized distance between the weight matrices of the current layer ℓ , denoted as Dist W , and the normalized distances between the post-activations of the previous layer ℓ -1 , denoted as Dist H . These distances are expressed as Dist W = dist ( vec ( W ( ℓ ) A ) , vec ( W ( ℓ ) B ) )

and Dist H = dist ( vec ( H ( ℓ -1) A ) , vec ( H ( ℓ -1) B ) ) , respectively. Figure 7 shows that for both spawning and permutation methods, Dist com is negligible compared with Dist W and Dist H , which confirms the commutativity condition. Note that we also rule out the trivial case where either the weight matrices or the post-activations in the two networks are already perfectly aligned, as weights and post-activations often differ significantly. We also add more baseline experiments for comparison (see Appendix B.3 for more results).

Additionally, we note that commutativity has a similarity to model stitching [19, 3]. In Appendix B.4, we show that a stronger form of model stitching (without an additional trainable layer) works for two networks that satisfy LLFC.

Conditions I and II Imply LLFC. Theorem 1 below shows that weak additivity for ReLU activations (Definition 3) and commutativity (Definition 4) imply LLFC.

Theorem 1 (Proof in Appendix A.2) . Given a dataset D , if two modes θ A and θ B satisfy weak additivity for ReLU activations (Definition 3) and commutativity (Definition 4), then

Note that the definition of LLFC (Definition 2) allows a scaling factor c , while Theorem 1 establishes a stronger version of LLFC where c = 1 . We attribute this inconsistency to the accumulation of errors 6 in weak additivity and commutativity conditions, since they are only approximated satisfied in practice. Yet in most cases, we observe that c is close to 1 (see Appendix B.2 for more results).

5.2 Justification of the Permutation Methods

In this subsection, we provide a justification of the permutation methods in Git Re-Basin [1]: weight matching (3) and activation matching (4). Recall from Section 3 that given two modes θ A and θ B , the permutation method aims to find a permutation π = { P ( ℓ ) } L -1 ℓ =1 such that the permuted θ ′ B = π ( θ B ) and θ A are linearly connected, where P ( ℓ ) is a permutation matrix applied to the ℓ -th layer feature. Concretely, with a permutation π , we can formulate W ′ B ( ℓ ) and H ′ B ( ℓ ) of θ ′ B in each layer ℓ as W ′ B ( ℓ ) = P ( ℓ ) W ( ℓ ) B P ( ℓ -1) ⊤ and H ′ B ( ℓ ) = P ( ℓ ) H ( ℓ ) B [1]. 7 In Section 5.1, we have identified the commutativity property as a key factor contributing to LLFC. The commutativity property (8) between θ A and θ ′ B can be written as

We note that the weight matching (3) and activation matching (4) objectives can be written as min π ∑ L ℓ =1 ∥ ∥ ∥ W ( ℓ ) A -P ( ℓ ) W ( ℓ ) B P ( ℓ -1) ⊤ ∥ ∥ ∥ 2 F and min π ∑ L -1 ℓ =1 ∥ ∥ ∥ H ( ℓ ) A -P ( ℓ ) H ( ℓ ) B ∥ ∥ ∥ 2 F , respectively, which directly correspond to the two factors in Equation (9). Therefore, we can interpret Git Re-Basin as a means to ensure commutativity. Extended discussion on Git Re-basin [1] can be found in Appendix B.5.

6 Conclusion and Discussion

We identified Layerwise Linear Feature Connectivity (LLFC) as a prevalent phenomenon that cooccurs with Linear Mode Connectivity (LMC). By investigating the underlying contributing factors to LLFC, we obtained novel insights into the existing permutation methods that give rise to LMC. The consistent co-occurrence of LMC and LLFC suggests that LLFC may play an important role if we want to understand LMC in full. Since the LLFC phenomenon suggests that averaging weights is roughly equivalent to averaging features, a natural future direction is to study feature averaging methods and investigate whether averaging leads to better features.

We note that our current experiments mainly focus on image classification tasks, though aligning with existing literature on LMC. We leave the exploration of empirical evidence beyond image classification as future direction. We also note that our Theorem 1 predicts LLFC in an ideal case, while in practice, a scaling factor c is introduced to Definition 2 to better describe the experimental results. Realistic theorems and definitions (approximated version) are defered to future research.

Finally, we leave the question if it is possible to find a permutation directly enforcing the commutativity property (9). Minimizing ∥ ( W ( ℓ ) A -P ( ℓ ) W ( ℓ ) B P ( ℓ -1) ⊤ )( H ( ℓ -1) A -P ( ℓ -1) H ( ℓ -1) B ) ∥ F entails solving Quadratic Assignment Problems (QAPs) (See Appendix A.3 for derivation), which are known NP-hard. Solving QAPs calls for efficient techniques especially seeing the progress in learning to solve QAPs [37, 34].

References

Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=CQsmMYmlP5T .

Stephen Ashmore and Michael Gashler. A method for finding similarity between multi-layer perceptrons by forward bipartite alignment. In 2015 International Joint Conference on Neural Networks (IJCNN) , pages 1-7. IEEE, 2015.

Yamini Bansal, Preetum Nakkiran, and Boaz Barak. Revisiting model stitching to compare neural representations. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021. URL https://openreview.net/forum?id=ak06J5jNR4 .

Yuan Cao, Zixiang Chen, Misha Belkin, and Quanquan Gu. Benign overfitting in two-layer convolutional neural networks. Advances in neural information processing systems , 35:25237-25250, 2022.

Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International conference on machine learning , pages 1309-1318. PMLR, 2018.

Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=dNigytemkL .

Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. Advances in Neural Information Processing Systems , 33:5850-5861, 2020.

Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations , 2019. URL https://openreview. net/forum?id=rJl-b3RcF7 .

Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning , pages 3259-3269. PMLR, 2020.

C. Daniel Freeman and Joan Bruna. Topology and geometry of half-rectified network optimization. In International Conference on Learning Representations , 2017. URL https://openreview.net/forum? id=Bk0FWVcgx .

Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems , 31, 2018.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770-778, 2016. doi: 10.1109/CVPR.2016.90.

Robert Hecht-Nielsen. On the algebraic structure of feedforward network weight spaces. 1990.

Tjalling C Koopmans and Martin Beckmann. Assignment problems and the location of economic activities. Econometrica: journal of the Econometric Society , pages 53-76, 1957.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Rohith Kuditipudi, Xiang Wang, Holden Lee, Yi Zhang, Zhiyuan Li, Wei Hu, Rong Ge, and Sanjeev Arora. Explaining landscape connectivity of low-cost solutions for multilayer nets. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_ files/paper/2019/file/46a4378f835dc8040c8057beb6a2da52-Paper.pdf .

Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N , 7(7):3, 2015.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equivariance and equivalence. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 991-999, 2015.

Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent learning: Do different neural networks learn the same representations? In Dmitry Storcheus, Afshin Rostamizadeh, and Sanjiv Kumar, editors, Proceedings of the 1st International Workshop on Feature Extraction: Modern Questions and Challenges at NIPS 2015 , volume 44 of Proceedings of Machine Learning Research , pages 196-212, Montreal, Canada, 11 Dec 2015. PMLR. URL https://proceedings.mlr.press/v44/ li15convergent.html .

Shiyu Liang, Ruoyu Sun, Yixuan Li, and Rayadurgam Srikant. Understanding the loss surface of neural networks for binary classification. In International Conference on Machine Learning , pages 2835-2843. PMLR, 2018.

Chang Liu, Chenfei Lou, Runzhong Wang, Alan Yuhan Xi, Li Shen, and Junchi Yan. Deep neural network fusion via graph matching with applications to model ensemble and federated learning. In ICML , pages 13857-13869, 2022. URL https://proceedings.mlr.press/v162/liu22k.html .

Ekdeep Singh Lubana, Eric J Bigelow, Robert P. Dick, David Krueger, and Hidenori Tanaka. Mechanistic mode connectivity, 2023. URL https://openreview.net/forum?id=NZZoABNZECq .

Vaishnavh Nagarajan and J Zico Kolter. Uniform convergence may be unable to explain generalization in deep learning. Advances in Neural Information Processing Systems , 32, 2019.

Quynh Nguyen. On connected sublevel sets in deep learning. In International conference on machine learning , pages 4790-4799. PMLR, 2019.

Quynh Nguyen, Mahesh Chandra Mukkamala, and Matthias Hein. On the loss landscape of a class of deep neural networks with no bad local valleys. In International Conference on Learning Representations , 2019. URL https://openreview.net/forum?id=HJgXsjA5tQ .

David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning internal representations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.

Thiago Serra, Xin Yu, Abhinav Kumar, and Srikumar Ramalingam. Scaling up exact neural network compression by reLU stability. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021. URL https://openreview.net/forum? id=tqQ-8MuSqm .

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. URL http://arxiv.org/abs/1409.1556 .

Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. Advances in Neural Information Processing Systems , 33:22045-22055, 2020.

Norman Tatro, Pin-Yu Chen, Payel Das, Igor Melnyk, Prasanna Sattigeri, and Rongjie Lai. Optimizing mode connectivity via neuron alignment. Advances in Neural Information Processing Systems , 33:1530015311, 2020.

Luca Venturi, Afonso S. Bandeira, and Joan Bruna. Spurious valleys in one-hidden-layer neural network optimization landscapes. J. Mach. Learn. Res. , 20:133:1-133:34, 2019. URL http://jmlr.org/ papers/v20/18-674.html .

Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id=BkluqlSFDS .

Runzhong Wang, Junchi Yan, and Xiaokang Yang. Neural graph matching network: Learning lawler's quadratic assignment problem with extension to hypergraph and multiple-graph matching. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning , pages 23965-23998. PMLR, 2022.

Zhiwei Xu, Yutong Wang, Spencer Frei, Gal Vardi, and Wei Hu. Benign overfitting and grokking in relu networks for xor cluster data. arXiv preprint arXiv:2310.02541 , 2023.

Tianshu Yu, Junchi Yan, Yilin Wang, Wei Liu, and baoxin Li. Generalizing graph matching beyond quadratic assignment model. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper_files/paper/2018/file/ 51d92be1c60d1db1d2e5e7a07da55b26-Paper.pdf .

David Yunis, Kumar Kshitij Patel, Pedro Henrique Pamplona Savarese, Gal Vardi, Jonathan Frankle, Matthew Walter, Karen Livescu, and Michael Maire. On convexity and linear mode connectivity in neural networks. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop) , 2022. URL https://openreview.net/forum?id=TZQ3PKL3fPr .

Zhenyu Zhu, Fanghui Liu, Grigorios Chrysos, Francesco Locatello, and Volkan Cevher. Benign overfitting in deep neural networks under lazy training. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 43105-43128. PMLR, 23-29 Jul 2023. URL https://proceedings.mlr.press/v202/zhu23h.html .

A Proofs of Theorems and Derivations

A.1 Proof of Lemma 1

In this section, we prove Lemma 1 in the main paper. This lemma indicates that we can directly imply Linear Mode Connectivity (LMC, see Definition 1) from Layerwise Linear Feature Connectivity (LLFC, see Definition 2) applied to last layer.

Definition 1 ( Linear Mode Connectivity ) . Given a test dataset D and two modes θ A and θ B such that Err D ( θ A ) ≈ Err D ( θ B ) , we say θ A and θ B are linearly connected if they satisfy

Definition 2 ( Layerwise Linear Feature Connectivity ) . Given dataset D and two modes θ A , θ B of an L -layer neural network f , the modes θ A and θ B are said to be layerwise linearly feature connected if they satisfy

Lemma 1. Suppose two modes θ A , θ B satisfy LLFC on a dataset D and

Proof. Note that the classification depends on the relative order of the entries in the output of the final layer. As a consequence, for each data point in the dataset D , the linear interpolation of the outputs of the models makes the correct classification if both models make the correct classification. Therefore, only if one of the model makes the incorrect classification, the linear interpolation of the outputs of the models would possibly make the incorrect classification, i.e,

Since θ A and θ B satisfy LLFC, then at last layer we have

then have

According to the condition that which indicates

and this finishes the proof.

A.2 Proof of Theorem 1

In this section, we prove Theorem 1 in the main paper. Theorem 1 indicates that we can derive LLFC from two simple conditions: weak additivity for ReLU activations (Definition 3) and commutativity (Definition 4). Note that though we consider a multi-layer perceptron (MLP) for convenience, our proof and results can be easily adopted to any feed-forward structure, e.g., a convolutional neural network (CNN).

Definition 3 ( Weak Additivity for ReLU Activations ) . Given a dataset D , two modes θ A and θ B are said to satisfy weak additivity for ReLU activations if

then we have

Definition 4 ( Commutativity ) . Given a dataset D , two modes θ A and θ B are said to satisfy commutativity if

Theorem 1. Given a dataset D , if two modes θ A and θ B satisfy weak additivity for ReLU activations (Definition 3) and commutativity (Definition 4), then

Proof. Before delving into the proof, let us denote the forward propagation in each layer ℓ by

Given θ A and θ B that satisfy the commutativity property, then ∀ ℓ ∈ [ L ] and ∀ α ∈ [0 , 1] , we have

Additionally, we can easily verify that

Subsequently,

Given the weak additivity for ReLU activation is satisfied for θ A and θ B , then we have

To conclude, ∀ ℓ ∈ [ L ] and ∀ α ∈ [0 , 1] , we have

For the right hand side of Equation (10), recursively, we can have

where X denotes the input data matrix.

Recall we denote H ( ℓ ) = f ( ℓ ) ( θ ; X ) which indicates

and this finishes the proof.

A.3 Derivation of Quadratic Assignment Problem

In this section, we aim to show that minimizing ∑ L ℓ =1 ∥ ∥ ∥ ( W ( ℓ ) A -P ( ℓ ) W ( ℓ ) B P ( ℓ -1) ⊤ )( H ( ℓ -1) A -P ( ℓ -1) H ( ℓ -1) B )∥ ∥ ∥ 2 F includes solving Quadratic Assignment Problems (QAPs), known to be NP-hard.

Consider its first term, i.e.,

Similarly, consider the second term, i.e,

which also gives rise to Koopmans-Beckmann's QAPs.

For the last term, i.e,

which entails solving bi-level matching problems.

Therefore, the objective can be rewritten as the summation of QAPs and bi-level matching problems and cannot be further simplified, which is NP-hard.

B More Experimental Details and Results

B.1 Detailed Experimental Settings

In this section, we introduce the detailed experimental setup. Before delving into details, recall that unless otherwise specified, in this paper we consider models trained on a training set, and then all the investigations are evaluated on a test set.

B.1.1 Spawning Method

Multi-Layer Perceptrons on the MNIST Dataset. In accordance with the settings outlined by Ainsworth et al. [1], we train multi-layer perceptron networks with three hidden layers, each consisting of 512 units, on the MNIST dataset. We adopt the ReLU activation between layers. Optimization is done with the Adam algorithm and a learning rate of 1 . 2 × 10 -4 . The batch size is set to 60 and the total number of training epochs is 30 . To find the modes that satisfy LMC, we start spawning from a common initialization θ (0) .

VGG-16 and ResNet-20 on the CIFAR-10 Dataset. In accordance with the settings outlined by Frankle et al. [9], we train the VGG-16 architecture [29] and the ResNet-20 architecture [12] on the CIFAR-10 dataset. Data augmentation techniques include random horizontal flips and random 32 × 32 pixel crops. Optimization is done using SGD with momentum (momentum set to 0.9). A weight decay of 1 × 10 -4 is applied. The learning rate is initialized at 0 . 1 and is dropped by 10 times at 80 and 120 epochs. The total number of epochs is 160 . To find the modes that satisfy LMC, we start spawning after training 5 epochs for both VGG-16 and ResNet-20.

ResNet-50 on the Tiny-ImageNet Dataset. In accordance with the settings outlined by Frankle et al. [9], we train the ResNet-50 architecture [12] on the Tiny-ImageNet dataset. Data augmentation techniques include random horizontal flips and random 32 × 32 pixel crops. Optimization is done using SGD with momentum (momentum set to 0.9). A weight decay of 1 × 10 -4 is applied. The learning rate is set to 0 . 4 and warmed up for 5 epochs and then is dropped by 10 times at 30 , 60 and 80 epochs. The total number of epochs is 90 . To find the modes that satisfy LMC, we start spawning after training 14 epochs.

B.1.2 Permutation Method

For the permutation method, we follow the experimental settings of Ainsworth et al. [1] strictly, which are described below.

Multi-Layer Perceptrons on MNIST and CIFAR-10. Similar to the spawning method, we use multi-layer perceptron (MLP) networks with three hidden layers, each consisting of 512 units. For MNIST, optimization is performed using Adam with a learning rate of 1 × 10 -3 . For CIFAR-10, optimization is performed using SGD with a learning rate of 0 . 1 . Both activation matching and weight matching are used to identify modes that satisfy LMC.

ResNet-20 on CIFAR-10. To achieve LMC, we modify the ResNet-20 architecture by incorporating LayerNorms in place of BatchNorms. Furthermore, we increase the width of ResNet-20 by a factor of 32. Data augmentation techniques include random horizontal flips, random 32 × 32 pixel crops, random resizes of the image between 0 . 8 × and 1 . 2 × , and random rotations between ± 30 ◦ . The optimization process involves using SGD with momentum (set to 0.9). A weight decay regularization term of 5 × 10 -4 is applied. A single cosine decay schedule with a linear warm-up is applied, where the learning rate is initialized to 1 × 10 -6 and gradually increased to 0 . 1 over the course of an epoch, and then a single cosine decay schedule is applied for the remaining training. Only weight matching is used to identify modes that satisfy LMC.

Unlike the spawning method, VGG models are not used in the permutation method due to their inability to achieve LMC. Additionally, Ainsworth et al. [1] open-sourced their source code and pretrained checkpoints. Therefore, we directly use the pre-trained checkpoints provided by Ainsworth et al. [1].

B.2 Verification of LLFC Co-Occuring with LMC

In this section, we provide extensive experimental results to verify that LLFC consistently co-occurs with LMC, and conduct a new experiment to demonstrate that the constant c is close to 1 in most cases. Both the spawning method and the permutation method are utilized to obtain linearly connected modes θ A and θ B . As shown in Figures 8 to 13 and 15, we include experimental results for MLP on the MNIST dataset (spawning method, activation matching, and weight matching), MLP on the CIFAR-10 dataset (both activation matching and weight matching), VGG-16 on the CIFAR-10 dataset (spawning method), ResNet-20 on the CIFAR-10 dataset (spawning method and weight matching) and ResNet-50 on the Tiny-ImageNet dataset (spawning method). In particular, in Figure 14, we include experimental results of Straight-Trough Estimator (STE) [1]. STE method tries to learn a permutation with STE that could minimize the loss barrier between one mode and the other permuted mode.

To verify the LLFC property on each data point x i in the test set D , we measure cosine α ( x i ) = cos[ f ( ℓ ) ( α θ A + (1 -α ) θ B ; x i ) , αf ( ℓ ) ( θ A ; x i ) + (1 -α ) f ( ℓ ) ( θ B ; x i )] . We compare this to the baseline cosine similarity cosine A,B ( x i ) = cos[ f ( ℓ ) ( θ A ; x i ) , f ( ℓ ) ( θ B ; x i )] . In Figures 8 to 15, we

coef 𝛼

= 𝑓 𝑙

cosine𝛼

= cos⟨𝑓 𝑙

cosine 𝐴.𝐵

1 - 𝛼 𝜃𝐵

, 𝛼𝑓

𝑙

𝜃𝐴

1 - 𝛼 𝜃𝐵

, 𝛼𝑓

𝑙

= cos 𝑓 𝑙

𝜃𝐴

, 𝑓

+ 1 - 𝛼 𝑓 𝑙

𝜃𝐴

𝑙

𝜃𝐵

𝜃𝐵

/||𝛼𝑓 𝑙

+ 1 - 𝛼 𝑓 𝑙 (𝜃 𝐵

𝜃𝐴

+ 1 - 𝛼 𝑓 𝑙

𝜃𝐵

||

2

MLP on MNIST

FC1

FC2

FC3

FC4

1 - cosine0.25

1 - cosine0.50

1 - cosine0.75

1 - cosine 𝐴,𝐵

FC1

FC2

FC3

FC4

coef0.50

coef0.75

coef0.25

0.5

工

工

0.0

1.0

工

工

工

工

工

工

工

工

0.5

0.0

1 - cosine0.25

1 - cosine0.50

1 - cosine0.75

1 - cosine 𝐴,𝐵

coef0.50

coef0.75

coef0.25

VGG16 on CIFAR10

Conv 1-1

Conv 1-2

Conv 2-1

Conv 2-2

Conv 3-1

Conv 3-2

Conv 3-3

Conv 4-1

FC 1

FC 2

FC 3

Conv 4-2

Conv 4-3

Conv 5-1

Conv 5-2

Conv 5-3

Conv 1-1

Conv 1-2

Conv 2-1

Conv 2-2

Conv 3-1

Conv 3-2

Conv 3-3

Conv 4-1

FC 1

FC 2

FC 3

Conv 4-2

Conv 4-3

Conv 5-1

Conv 5-2

Conv 5-3

工

0.5

工

工

工

工

工

工

工

0.0

.0

工

0.5

工

工

0.0

工工

工

工

工工

工工工

III

工工工

工工工工

1.0

0.5

0.0

1.0

0.5

0.0

conclude that the values of E D [1 -cosine α ( x i )] are close to 0 compared with E D [1 -cosine A,B ( x i )] , and thus verify our claim.

To show that the constant c is close to 1 in most cases, for each data point x i in the test set D , we measure coef α ( x i ) = ∥ f ( ℓ ) ( α θ A + (1 -α ) θ B ; x i ) ∥ cosine α ( x i ) / ∥ αf ( ℓ ) ( θ A ; x i ) + (1 -α ) f ( ℓ ) ( θ B ; x i )] ∥ , where ∥ f ( ℓ ) ( α θ A +(1 -α ) θ B ; x i ) ∥ cosine α ( x i ) denotes the length of f ( ℓ ) ( α θ A + (1 -α ) θ B ; x i ) projected on αf ( ℓ ) ( θ A ; x i ) + (1 -α ) f ( ℓ ) ( θ B ; x i )] . In Figures 8 to 15, we conclude that the values of E D [coef α ( x i )] are close to 1 in most cases, and thus verify our claim.

coef 𝛼

= 𝑓 𝑙

cosine𝛼

= cos⟨𝑓 𝑙

cosine 𝐴.𝐵

1 - 𝛼 𝜃𝐵

, 𝛼𝑓

𝑙

𝜃𝐴

1 - 𝛼 𝜃𝐵

, 𝛼𝑓

𝑙

= cos 𝑓 𝑙

𝜃𝐴

, 𝑓

+ 1 - 𝛼 𝑓 𝑙

𝜃𝐴

𝑙

𝜃𝐵

𝜃𝐵

/||𝛼𝑓 𝑙

+ 1 - 𝛼 𝑓 𝑙 (𝜃 𝐵

𝜃𝐴

+ 1 - 𝛼 𝑓 𝑙

𝜃𝐵

||

2

1 - cosine0.25

1 - cosine0.50

1 - cosine0.75

1 - cosine 𝐴,𝐵

coef0.50

coef0.75

coef0.25

ResNet20 on CIFAR10

Block 2-3

Block 3-1

Block 3-2

Block 3-3

FC

Block 1-1

Block 1-2

Block 1-3

Block 2-1

Block 2-2

Block 2-3

Block 3-1

Block 3-2

Block 3-3

FC

Block 1-1

Block 1-2

Block 1-3

Block 2-1

Block 2-2

0.5

0.0

1.0

工

工

工

五工工

0.5

0.0

1 - cosine0.25

1 - cosine0.50

1 - cosine0.75

1 - cosine 𝐴,𝐵

coef0.50

coef0.75

coef0.25

Block 1-1

Block 1-2

Block 1-3

Block 2-1

Block 2-2

Block 2-3

Block 2-4

Block 3-1

Block 4-1

Block 4-2

Block 4-3

𝐸𝐷[1 - cosine 𝒙𝑖

]

Block 3-2

Block 3-3

Block 3-4

Block 3-5

Block 3-6

𝐸𝐷[1 - cosine 𝒙𝑖

]

Block 1-1

Block 1-2

Block 1-3

Block 2-1

Block 2-2

Block 2-3

Block 2-4

Block 3-1

Block 4-1

Block 4-2

Block 4-3

𝐸𝐷[coef 𝒙𝑖

]

Block 3-2

Block 3-3

Block 3-4

Block 3-5

Block 3-6

𝐸𝐷[coef 𝒙𝑖

]

ResNet50 on Tiny-ImageNet

0.0

0.5

0.0

1.0

0.5

0.0

1.0

0.5

0.0

B.3 Verification of Commutativity

In this section, we provide more experimental results on various datasets and model architectures to verify the commutativity property for modes that satisfy LLFC. As shown in Figures 16 to 18, we include more experiments results for VGG-16 on the CIFAR-10 dataset (spawning method), MLP on the MNIST dataset (activation matching) and MLP on the CIFAR-10 dataset (both activation matching and weight matching).

coef 𝛼

= 𝑓 𝑙

cosine 𝐴.𝐵

1 - 𝛼 𝜃𝐵

, 𝛼𝑓

𝑙

𝜃𝐴

= cos 𝑓 𝑙

𝜃𝐴

, 𝑓

+ 1 - 𝛼 𝑓 𝑙

𝑙

𝜃𝐵

𝜃𝐵

/||𝛼𝑓 𝑙

𝜃𝐴

+ 1 - 𝛼 𝑓 𝑙

𝜃𝐵

||

2

FC1

FC2

FC3

FC4

coef0.50

coef0.75

coef0.25

MLP on MNIST (Activation Matching)

FC1

FC2

FC3

FC4

1 - cosine0.25

1 - cosine0.50

1 - cosine0.75

1 - cosine 𝐴,𝐵

MLP on MNIST (Weight Matching)

FC1

FC2

FC3

FC4

FC1

FC2

FC3

FC4

0.5

工

士

工

0.0

工

0.5

工

0.0

1.0

工

工

工

工工

工

工

工

工

T

工

0.5

0.0

1.0

0.5

工

工

工

工

工

工

工

工

工

工

工

0.0

MLP on CIFAR10 (Activation Matching)

FC1

FC2

FC3

FC4

1 - cosine0.25

1 - cosine0.50

1 - cosine0.75

1 - cosine 𝐴,𝐵

FC1

FC2

FC3

FC4

coef0.50

coef0.75

coef0.25

MLP on CIFAR10 (Weight Matching)

FC1

FC2

FC3

FC4

FC1

FC2

FC3

FC4

0.5

工

0.0

工

工

工工工工

0.5

工

0.0

工

1.0

工

工

工

工

工

0.5

0.0

1.0

工

工

工

工

0.5

0.0

To verify the commutativity generally holds for modes that satisfy LLFC, for test set D , we compute Dist com = dist ( vec ( W ( ℓ ) A H ( ℓ -1) A + W ( ℓ ) B H ( ℓ -1) B ) , vec ( W ( ℓ ) A H ( ℓ -1) B + W ( ℓ ) B H ( ℓ -1) A ) ) 8 . Furthermore, we compare Dist com with Dist W = dist ( vec ( W ( ℓ ) A ) , vec ( W ( ℓ ) B ) ) and Dist H = dist ( vec ( H ( ℓ -1) A ) , vec ( H ( ℓ -1) B ) ) , respectively. In Figures 16 to 18, Dist com is negligible compared with Dist W and Dist H , confirming the commutativity condition.

Furthermore, we add baselines of models that are not linearly connected to further validate the commutativity condition. In Figure 19, we include experimental results for ResNet-20 on CIFAR-10 dataset (both spawning and weight matching method). Specifically, we measure Dist com,LMC of two linearly connected modes and Dist com,not LMC of two independently trained modes. In Figure 19, the values of Dist com,LMC are negligible compared with Dist com,not LMC , which confirms the commutativity condition.

coef 𝛼

= 𝑓 𝑙

cosine 𝐴.𝐵

1 - 𝛼 𝜃𝐵

, 𝛼𝑓

𝑙

𝜃𝐴

= cos 𝑓 𝑙

𝜃𝐴

, 𝑓

+ 1 - 𝛼 𝑓 𝑙

𝑙

𝜃𝐵

𝜃𝐵

/||𝛼𝑓 𝑙

𝜃𝐴

+ 1 - 𝛼 𝑓 𝑙

𝜃𝐵

||

2

MLP on MNIST (STE)

𝐸𝐷[1 - cosine 𝒙𝑖

]

FC1

FC2

FC3

FC4

1 - cosine0.25

1 - cosine0.50

1 - cosine0.75

1 - cosine 𝐴,𝐵

𝐸𝐷[coef 𝒙𝑖

]

FC1

FC2

FC3

FC4

coef0.50

coef0.75

coef0.25

MLP on CIFAR10 (STE)

𝐸𝐷[1 - cosine 𝒙𝑖

]

FC1

FC2

FC3

FC4

𝐸𝐷[coef 𝒙𝑖

]

FC1

FC2

FC3

FC4

0.5

工

工

0.0

0.5

工

0.0

工

工

工

工

工工工工

1.0

工

工

工

工

0.5

0.0

1.0

0.5

工

工

工

工

工

工

工

工

工

工

工

0.0

1 - cosine0.25

1 - cosine0.50

1 - cosine0.75

1 - cosine 𝐴,𝐵

coef0.50

coef0.75

coef0.25

Block 2-3

Block 3-1

Block 3-2

Block 3-3

FC

Block 1-1

Block 1-2

Block 1-3

Block 2-1

Block 2-2

Block 2-3

Block 3-1

Block 3-2

Block 3-3

FC

Block 1-1

Block 1-2

Block 1-3

Block 2-1

Block 2-2

ResNet20 (32 x) on CIFAR10 (Weight Matching)

0.5

0.0

1.0

工

工

工

工

工

工

工

工

0.5

0.0

Conv 1-2

Distance

Conv 4-1

Conv 4-2

Distance

Distance

Conv 2-2

Distance

Conv 3-1

Distance

Conv 2-1

Distance

Distance

Conv 5-2

Distance

Conv 5-3

Distance

Conv 5-1

Conv 4-3

Distance

Conv 3-2

Conv 3-3

Distance

Distance

VGG16 on CIFAR10

Dist 𝑐𝑜𝑚

Dist 𝑊

Dist 𝐻

Dist 𝑐𝑜𝑚 =

Dist 𝑐𝑜𝑚

Dist 𝑊

Dist 𝐻

Distance

Distance

Distance

FC 2

FC 3

FC 4

MLP on MNIST (Activation Matching)

Verification of Commutativity

Version 2

Dist 𝑐𝑜𝑚

Dist 𝑊

Dist 𝐻

FC 2

FC 3

FC 4

Distance

Distance

Distance

MLP on CIFAR10 (Activation Matching)

FC 2

FC 3

FC 4

Distance

Distance

Distance

MLP on CIFAR10 (Weight Matching)

Dist 𝑐𝑜𝑚 =

|| 𝑊1

𝑙 𝐻1

𝑙-1

+𝑊2 𝑙 𝐻

2

𝑙-1

-(𝑊1 𝑙 𝐻2

𝑙-1

+𝑊2 𝑙 𝐻

1

𝑙-1

)||

2

||𝑊 1

𝑙 𝐻1

𝑙-1

+𝑊2 𝑙 𝐻

2

𝑙-1

|| ||𝑊

1

𝑙 𝐻2

𝑙-1

+𝑊2 𝑙 𝐻

1

𝑙-1

||

Dist 𝑊 =

||𝑊 1

𝑙

-𝑊2 𝑙 ||

2

||𝑊 1

𝑙

|| ||𝑊

2

𝑙

||

Dist 𝐻 =

||𝐻 1

𝑙-1 -𝐻2

𝑙-1

||

2

||𝐻

1

𝑙-1

|| ||𝐻

2

𝑙-1

||

FC

Block 2-3

Block 3-1

Block 3-2

Block 3-3

FC

Block 2-3

Block 3-1

Block 3-2

Block 3-3

ResNet20 on CIFAR10 (Spawning)

ResNet20 (32 x) on CIFAR10 (Weight Matching)

Dist 𝑐𝑜𝑚, 𝐿𝑀𝐶

Dist 𝑐𝑜𝑚, 𝑛𝑜𝑡 𝐿𝑀𝐶

Distance

Distance

Distance

Distance

Distance

Distance

Distance

Distance

Distance

Distance

|| 𝑊1

𝑙 𝐻1

||𝑊 1

𝑙-1

+𝑊2 𝑙 𝐻

𝑙-1

𝑙 𝐻1

2

𝑙-1

+𝑊2 𝑙 𝐻

2

𝑙-1

-(𝑊1 𝑙 𝐻2

|| ||𝑊

1

𝑙 𝐻2

1

||𝑊 1

2

||𝐻 1

𝑙-1 -𝐻2

||𝐻

1

𝑙-1

𝑙-1

|| ||𝐻

2

||

𝑙-1

2

||

Notably, the experiments are not conducted on the first Conv/Linear layer of the model because the commutativity condition is naturally satisfied for the first layer where H (0) A = H (0) B = X where X is the input data matrix.

B.4 Experiments on Model Stitching

Model stitching [19, 3] is commonly employed to analyze neural networks' internal representations. Let A and B represent neural networks with identical architectures. Given a loss function L , model stitching involves finding a stitching layer s (e.g., a linear 1 × 1 convolutional layer) such that the minimization of L ( B >ℓ ◦ s ◦ A ≤ ℓ ) is achieved. Here, B >ℓ denotes the mapping from the activations of the ℓ -th layer of network B to the final output, A ≤ ℓ denotes the mapping from the input to the activations of the ℓ -th layer of network A , and ◦ represents function composition.

In this section, we explore a stronger form of model stitching. Specifically, given two neural networks A and B that satisfy LLFC, we evaluate the arruacy of B >ℓ ◦ A ≤ ℓ over the test set D without finding a stitching layer, i.e., Err D ( B >ℓ ◦ A ≤ ℓ ) . As shown in Tables 1 to 3, we include experimental results for MLP on the MNIST dataset, VGG-16 on CIFAR-10 the dataset and ResNet-20 on the CIFAR-10 dataset. Only the spawning method is utilized to find modes that satisfy LLFC. The results depicted

Dist 𝐻 =

Singular values

Index of

𝑠

Index of

𝑠

Index of

𝑠

Singular values

Singular values

Block 3-1 Conv 1

Singular values

Index of

𝑠

Index of

𝑠

Index of

𝑠

Singular values

Singular values

Singular values

Index of

𝑠

Index of

𝑠

Index of

𝑠

Singular values

Singular values

FC 1

FC 2

FC 3

MLP on CIFAR10

ResNet20 (32 x) on CIFAR10

Block 3-3 Conv 2

Block 3-1 Conv 2

Block 3-2 Conv 1

Block 3-2 Conv 2

Block 3-3 Conv 1

in Tables 1 to 3 demonstrate that the error rates of the stitched model on the test set closely resemble the error rates of the original models A and B , regardless of the dataset or model architecture. This observation suggests that models that satisfy LLFC encode similar information, which can be decoded across different models. Subsequently, the experiments of model stitching provides new insights towards the commutativity property, i.e, ∀ ℓ ∈ [ L ] , W ( ℓ ) B H ( ℓ -1) A ≈ W ( ℓ ) B H ( ℓ -1) B .

B.5 Discussion on Git Re-basin [1]

In this section, we investigate the ability of permutation methods to achieve LMC. While we have interpreted the activation matching and weight matching methods proposed by Ainsworth et al. [1] as guaranteeing the commutativity property, we have yet to address why permutation methods can ensure the satisfaction of this property. Thus, in order to delve into the capability of permutation methods, we must address the question of why these methods are capable of ensuring the satisfaction of the commutativity property.

Low-rank model weights and activations contribute to ensure the commutativity property. We now consider a stronger form of the commutativity property, where given two modes θ A and θ B and a

Singular values

Index of

𝑠

Index of

𝑠

Index of

𝑠

Singular values

Singular values

FC 1

FC 2

FC 3

MLP on CIFAR10

ResNet20 (32 x) on CIFAR10

srank(𝑾 𝑻

(ℓ)

)

srank(𝑾 𝟏

(ℓ)

)

Block 3-1 Conv1

Block 3-3 Conv2

Block 3-1 Conv2

Block 3-2 Conv1

Block 3-2 Conv2

Block 3-3 Conv1

dataset D , we have:

Thus, to satisfy the commutativity property for a given layer ℓ , we can employ the permutation method to find a permutation matrix P ( ℓ -1) such that:

𝛽 denotes the smallest principal angle between spaces spanned by top 5% column vector of 𝑈 In a homogeneous linear system WX = 0 , a low-rank matrix W allows for a larger solution space for X . Therefore, if the ranks of W ( ℓ ) A and W ( ℓ ) B are low, it becomes easier to find a permutation matrix P ( ℓ -1) that satisfies the commutativity property. Similarly, if we consider another form of commutativity property:

Then, to ensure the commutativity property, we need to find P ( ℓ -1) and P ( ℓ ) such that

Then, if the ranks of H ( ℓ -1) A and H ( ℓ -1) B are low, it is easier to find the permutation matrices to satisfy the condition. In real scenarios, both model weights (see Figure 20) and activations (see Figure 21) are approximately low-rank, which helps the permutation methods satisfy the commutativity property.

Additionally, Ainsworth et al. [1] mentioned two instances where permutation methods can fail: models with insufficient width and models in the early stages of training. In both cases, the model weights often fail to satisfy the low-rank model weight condition. In the first scenario, when the model lacks sufficient width, meaning that the dimension of the weight matrix approaches the rank of the weight matrix, the low-rank condition may not be met. For example, compared the singular values of ResNet-20 (32x) (see Figure 20) with singular values of ResNet-20 (1x) (see ?? ), it is evident that in the wider architecture, the proportion of salient singular values is smaller. In the second scenario, during the initial stages of training, the weight matrices resemble random matrices and may not

exhibit low-rank characteristics. For example, as shown in Figure 22, the stable ranks of weight matrices of the model after convergence are significantly smaller than those of the model in the early stage of training. Consequently, permutation methods may struggle to find suitable permutations that fulfill the commutativity property, resulting in the inability to obtain modes that satisfy LMC.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThere is a continuing effort to understand the complex training dynamics and loss landscape of neural networks, and one of the most interesting discoveries is Linear Mode Connectivity (LMC). LMC is the phenomenon that when two different solutions are linearly interpolated in parameter space, the training loss and test loss remain low enough to be similar to the solution. In this paper, the authors propose Layerwise Linear Feature Connectivity (LLFC), a stronger linear connectivity than LMC. LLFC means that the feature maps of all layers of two differently trained solutions are also linearly connected. Through various experiments, they show that LLFC is satisfied when spawning and permutation methods, which are common methods for presenting LMC characteristics, are used. In addition, they advance the understanding of LMC with a theoretical explanation of the underlying factors that make LLFC appear naturally.\n\nSTRENGTHS:\n- The paper introduced LLFC, an interesting phenomenon that extends LMC, linear connectivity in parameter space, to linear connectivity in feature space.\n- The empirical results are comprehensive and contain the necessary contents for the development of the discussion.\n- The theoretical analysis provides a convincing explanation for the emergence of LLFC.\n\nWEAKNESSES:\n- The authors only ran their experiments on MNIST and CIFAR10, which are relatively easy datasets. It will be important to verify its validity on larger datasets such as ImageNet. Git Re-Basin (Ainsworth at el., 2023) also showed a relatively large loss barrier on ImageNet, so validation on a wider range of datasets is required.\n-----\n(Ainsworth at el., 2023) [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://openreview.net/forum?id=CQsmMYmlP5T)\n\nQUESTIONS:\n- I'm not sure how low the value of the sparsity measure is in Figure 4. Is a value of ~0.3 considered sparse?\n- In the experiments, the values measured with random variables, etc. are used as baseline, but it would be more accurate to compare them to the values for a model that is not linearly connected. The baseline values currently presented seem to exaggerate the results because they are comparing too different components.\n- Just a question, do you think the ensemble performance would be improved when ensembling linearly interpolated features with the modes?\n\nLIMITATIONS:\nI see no potential negative societal impact from this paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work identifies layerwise  linear feature connectivity (LLFC) and proves LLFC is sufficient for linear mode connectivity (LMC) between neural networks. Experimental evidence using two methods for finding LMC networks (spawning and permutation) finds that LMC and LLFC co-occur in a variety of models and datasets. Two conditions (weak additivity of ReLU and commutativity) are identified as sufficient for LLFC, and experimental evidence is presented for these conditions occuring in LMC networks.\n\nSTRENGTHS:\nThe paper unifies two lines of research into LMC (spawning versus permutation methods) under a common mathematical framework. The notion of commutativity is an interesting derivation and not as obvious as it seems at first glance. The paper is well written and easy to follow, with clear definitions. If established comprehensively, LLFC would be a significantly stronger condition than LMC, and would have far-reaching implications.\n\nWEAKNESSES:\nMain issues:\n- Current alignment algorithms for permuting LMC networks already directly optimize for LLFC (section 5.3), so the finding that LLFC implies LMC (Lemma 1) is formalizing a well-established phenomenon which is somewhat obvious.\n- As a result, LLFC implying LMC is somehow less interesting than LMC implying LLFC, but the latter direction is not explored. Only co-occurence is observed experimentally, making the direction of causation unclear. In particular, if LMC is found to imply LLFC, the reasons are likely to be extremely informative.\n- For spawning LMC networks, it is unclear how the early training epochs contribute to enabling LMC. The experiments do not consider the evolution of LMC networks through training time. Furthermore, the connection between figure 6 and the equation in lines 285-286 is unclear (see questions below).\n\nAdditionally, the experimental evidence lacks baselines for comparison. In general, a fair comparison should include interpolated non-LMC networks.\n- Figure 2 and 3: no cosine similarity for interpolated networks that are not linearly connected. Since interpolating between any vectors (including random ones) increases their cosine similarity considerably, it is not clear that the increase in similarity is due to LLFC and not averaging.\n- Figure 4: no sparsity baseline for random networks. Currently it is unclear what is considered \"small\" in line 232.\n- Figure 5: no commutativity distance for non-LMC networks\n\nQUESTIONS:\n- Section 5.2: how  does similarity of $U_A$ and $U_B$ imply commutativity when $W = U \\Sigma V$ also depends on $V$, which could differ significantly between two networks? The connection between figure 6 and the equation in lines 285-286 is unclear.\n- How much would solving the quadratic assignment problem improve over existing algorithms? Given that activation similarity is closely correlated with weight similarity, it is not obvious whether minimizing the QAP will lead to significant improvements in permutation alignment versus aligning weights and/or alignments, which is equivalent to minimizing one or both sides of equation 9 separately.\n- Weak additivity seems to rely heavily on the ReLU function. Does LLFC occur for other activation functions?\n- Do networks exist where LMC holds but not LLFC? If so, how can one find such networks and how common are they?\n\nLIMITATIONS:\nOverall the findings could be explored in greater depth and better related to observable phenomena. While LLFC is shown to imply LMC, the reverse is not shown or refuted. For spawned LMC networks, it is not clear whether LLFC is a byproduct of averaging, how LLFC relates to the onset of LMC at an early point in training, or how LLFC can be increased/decreased. For permuted LMC networks, current alignment algorithms (e.g. Ainsworth et al. 2022, Singh & Jaggi 2020) are already explicitly optimizing for LLFC (via weight or activation alignment), and thus the findings have limited explanatory power.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Layerwise Linear Feature Connectivity (LLFC). Compared to the better known linear mode connectivity (LMC), which states that networks trained by SGD are linearly connected modulo permutation, LLFC suggests that the feature maps of every layer is connected. As shown in the paper, LLFC is a strictly stronger property than LMC. Empirical results on a range of architectures (ResNet-20, VGG-16, MLP) and datasets (MNIST, CIFAR-10) suggest that LLFC co-occurs with LMC. The authors then identify two conditions that collectively imply LLFC: weak additivity, which requires ReLU to behave like a linear activation on two modes, and commutativity, which requires the next-layer linear transformations applied to the internal features of two networks can be interchanged. They verify that these two properties holds for modes that satisfy LLFC empirically. Finally, the authors show that two common methods to obtain linearly connected modes, the spawning method and the permutation method, both promote the commutativity property, which explains their effectiveness.\n\nSTRENGTHS:\n- The paper reveals a novel and more general notion of linear mode connectivity, which is an interesting phenomenon that attracts recent attention in the ML community. The authors provide precise definitions and a set of sufficient conditions for LLFC. Their observation is novel and advances the understanding of the origin of linear connectivity. \n- Experiments are sound and provide useful insights. In particular, empirical results support the occurrence of LLFC and validates that the two conditions for LLFC approximately hold in common settings. \n- Through dissecting the conditions required for LLFC, this paper also explains why the spawning method and permutation method produces LMC. Since theoretical results are sparse in explaining the origin of LMC, this paper’s contribution on the topic is significant. \n- The writing is clear and well organized.\n\nWEAKNESSES:\n- The analysis for the cause of LLFC is limited to ReLU activation. Extending the weak additivity condition to different activations does not seem straightforward. To demonstrate the prevalence of LLFC, it might help to include experiments on neural networks with different activations. \n- As pointed out by the authors and shown in figure 4 and 6, weak additivity for ReLU activations and commutativity only approximately hold in real neural networks. Hence Theorem 1 describes conditions for perfect LLFC in an idealistic setting. While this result is significant and novel, a more careful treatment of the approximated version could make the theoretical contribution stronger. \n- There are some gaps between theory and empirical observation. Specifically, the following results are not well explained: (a) Why are the pre-activations sparse? Is this universal to all architectures? (b) Why do the modes obtained by the spawning method share similar principal directions of model weights in each layer? While these are empirically verified in the paper, it would be helpful to point the readers to existing theoretical analysis, if available. \n- While LLFC is an interesting observation and provides insights on LMC, there seems to be few applications that leverages LLFC. The idea of averaging features mentioned in the conclusion is interesting, but it is not clear how to implement feature averaging and under what situations this would be beneficial.\n\nQUESTIONS:\n- In the definition of LLFC, there is a scaling factor $c$ that is not predicted by theorem 1. According to the authors, this inconsistency can be attributed to the accumulation of errors in the two conditions (line 263). Why does the accumulation of errors result in a scaling difference instead of another type of modification such as an additive term? \n- Does spawning or permutation have any impact on the sparsity of pre-activation $\\tilde{H}^{(l)}$ and the weak additivity condition? \n- Is condition 1+2 the only way to guarantee LLFC? In particular, identifying other sets of sufficient conditions for LLFC may lead to new permutation methods that find linearly connected modes, other than weight matching or activation matching.\n- The experiments use SGD with momentum in training. Does the choice of optimization method affect the occurrence of LLFC?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations. There are no potential negative societal impacts of this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the property of Layerwise Linear Feature Connectivity (LLFC) of neural network representations is introduced, which is a stronger generalization of linear mode connectivity (LMC). They show that LLFC often occurs when LMC does. Moreover, they give a possible mechanism by which LLFC may occur (ReLU activation additivity and weight commutatitivty), and provide evidence that LLFC often does occur by this mechanism. Finally, they reinterpret spawning and permutation-finding based methods for LMC as promoting commutativity.\n\nSTRENGTHS:\n1. Introduction exposition is quite good.\n2. The relationship between spawning / permutation-finding methods and commutatitivity is really interesting and insightful. The fact that both the weight-matching and activation-matching losses can be found in the commutativity condition is quite neat, as (to the best of my knowledge) these two methods were proposed as heuristics. The discussion of weight matrix rank in Appendix E is also insightful.\n3. Figure 5 on the empirical measurement of commutativity is very nice, especially because the two baseline numbers (weight distance and activation distance) are so large, so commutatitivity really seems to be a good thing to look at here.\n4. Although LLFC is stronger than LMC, in some sense it may make the study of LMC easier, because you have more to look at (features in each layer and the two sufficient conditions you give, not just loss of a whole neural net).\n\nWEAKNESSES:\n1. You don't really cover the last method (straight-through estimator) from Git Re-Basin, even though it works the best in many cases. Perhaps you should note this.\n2. Although these experimental setups are standard, it would be good to see to what extent this holds beyond these few architectures for image classification.\n\nQUESTIONS:\nTypos:\n1. In 5.2, (i) should have $W_{B, \\mathrm{pri}}^{(l)} H^{(l-1)}_A + W_{A, \\mathrm{pri}}^{(l)} H_B^{(l-1)}$.\n2. 5.3: \"conprehensive\" -> \"comprehensive\"\n\nLIMITATIONS:\nI do not see much discussion of limitations, besides on the hardness of exactly minimizing the commutativity property objective. Perhaps you could state that the empirical evidence is limited to image classification.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a special case of Linear Mode Connectivity (LMC) denoted as Layerwise Linear Feature Connectivity (LLFC). Whereas two trained neural networks present LMC if a convex combination of their parameters produce a neural network with similar training loss and accuracy, two trained neural networks present LLFC if a convex combination of their features in every layer also produces a neural network with similar features. Although being a special case, the authors find out that LMC and LLFC co-occur very often. Moreover, the authors characterize conditions that jointly imply LLFC between two ReLU networks.\n\n*****\n\nFollowing the authors' response, I am updating my score from 6 to 7.\n\nSTRENGTHS:\nThe writing of the paper is very clear and the experiments are distributed in such a way along the text that it is easy to follow along.\n\nMoreover, I find the study of how this conditions may emerge in ReLU networks particularly relevant, since in that simpler setting it is easier to understand what is going on.\n\nWEAKNESSES:\nThe authors sell the idea of LLFC in a very positive tone, which is actually quite common in papers, but I cannot help but wonder about the following: \n\nTwo neural networks presenting LLFC would have very similar parameters, and the fact that LLFC co-occurs with LMC implies that LMC is observed due to the similarity between trained neural networks - either because the first epochs of training determined what model would be ultimately obtained; or because there are relatively few optimal neural networks upon permutation.\n\nHowever, even under this interpretation, I believe that this study helps demystifying LMC if it turns out that LMC rarely occur without LLFC.\n\nQUESTIONS:\nThe introduction attributes the first observation of LMC to [8], but Section 2 attributes it to earlier work [21]. \n\nDefinition 2 needs rework because it is clearly invalid if $\\alpha=0$ or $\\alpha=1$ unless $c=1$. In fact, this is not even how you measure it in Section 4, since you use cosine similarity instead. I believe that the use of a tolerance factor  $\\epsilon$ like you did in Line 220 would be more adequate in this definition.\n\nWould it make sense to assume that Definition 3 has some connection with stable neurons in ReLU networks, as in the following papers: https://arxiv.org/abs/1711.07356 & https://arxiv.org/abs/2102.07804 ?\n\nIt seems to me that Condition 1 was inspired by spawning and Condition 2 by permutation, even though they are not worded in that way. Is that a valid way to interpret them?\n\nI would like to hear the thoughts of the authors about what I described in Weaknesses.\n\nLIMITATIONS:\nI could not identify a discussion about limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}