Going Beyond Linear Mode Connectivity: TheLayerwise Linear Feature Connectivity
Zhanpeng Zhou1, Yongyi Yang2, Xiaojiang Yang1, Junchi Yan1*, Wei Hu2
1Shanghai Jiao Tong University, 2University of Michigan*Corresponding author

Background: Linear Mode Connectivity
Linear Mode Connectivity (LMC)
Given dataset and two modes ,thatsatisfy the linear mode connectivity if*Err*, two modeand(𝜽) denotes the classification error of the network 𝑓 𝜽; ⋅ on the dataset 𝐷.

Frankle et al. [1] observed LMC for
networks that are jointly trained fora short time before independent
training (spawning method).
Fig. 1: Illustration of spawning method and LMC [1].
[1] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis.

Background: Permutation Method
Permutation Invariance.
Given an -layer MLP , we can permute the neurons of the MLP in each layerℓ*):
without changing its functionality (arepermutation
matricesℓ∈[ ](ℓ)ℓℓ∈ 𝑳(ℓ)*Note that 𝑃(ℓ)

(ℓ)

(ℓ)

(ℓ) (ℓ)

ℓ∈[𝑳]
(ℓ)

(ℓ)

(ℓ)

and 𝑃 ( ) are all fixed to be identity matrix.

Independently trained networks can be linearly connected when considering
permutation invariance (permutation methods)[2, 3].
[2] Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks.
[3] Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries.

Background: Permutation Method
Ainsworth et al. [3] proposed weight matching and activation matching to achieve LMC:(ℓ)

weight matching∗: min𝑾(ℓ)−𝑷 ℓ 𝑾

𝑷ℓℓ

Activation matching∗: min𝑯(ℓ)(ℓ)−𝑷 ℓ 𝑯ℓ*We denote ℓ-th layer feature as 𝑯(ℓ) over the dataset 𝐷. Subscript

𝐴, 𝐵 corresponds to modes 𝜽 , 𝜽 .

Fig. 2: Illustration of permutation [2].

[3] Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries.

Motivation𝑓 ℓ (𝜽 ; 𝑿)𝑓 ℓ (𝛼𝜽 + 1 − 𝛼 𝜽 ; 𝑿)∗

Relationship?

𝑓 ℓ (𝜽 ; 𝑿)

what happens to the internal features when we linearlyinterpolate the weights of two trained networks?
*𝑓 ℓ (𝜽) denotes ℓ-th layer feature of the network 𝑓 𝜽; ⋅over the dataset 𝐷.

Layerwise Linear Feature Connectivity
Layerwise Linear Feature Connectivity (LLFC)
Given dataset and two modes ,of an -layer neural network , the modes
andare layerwise linearly feature connected if:∀ℓ ∈ 𝐿 , ∀𝛼 ∈ 0, 1 , ∃𝑐 > 0, 𝑠. 𝑡. , 𝑐𝑓 ℓ 𝛼𝜽 + 1 − 𝛼 𝜽𝑓 ℓ (𝛼𝜽 + 1 − 𝛼 𝜽 ; 𝑿)𝑓 ℓ (𝜽 ; 𝑿)

= 𝛼𝑓 ℓ 𝜽+ 1−𝛼 𝑓 ℓ 𝜽 .

𝑓 ℓ (𝜽 ; 𝑿)

Layerwise Linear Feature Connectivity
LLFC always co-occurs with LMC in practiceFig. 3: Comparison of 𝐸 [1 − cosine (𝒙 )]* and 𝐸 [1 − cosine , (𝒙 )]*, 𝛼 ∈ {.25, .5, . 75}.

Lemma (LLFC implies LMC)
Two modes ,satisfy LLFC over dataset*cosine𝒙and= cos⟨𝑓 ℓ 𝛼𝜃 + 1 − 𝛼 𝜃 ; 𝒙 , 𝛼𝑓 ℓ 𝜃 ; 𝒙 + 1 − 𝛼 𝑓 ℓ (𝜃 ; 𝒙 )⟩ and cosine , 𝒙= cos⟨𝑓 ℓ 𝜃 ; 𝒙 , 𝑓 ℓ (𝜃 ; 𝒙 )⟩Why LLFC Emerges?
Two simple conditions that leads to LLFC.
Condition I: Weak Additivity for ReLU Activations
Given dataset , the modes
andsatisfy weak additivity for ReLU activations ifℓ*We denote ℓ-th layer pre-activations asℓℓℓ

𝑯(ℓ) over the dataset 𝐷 and ReLU activation as 𝜎(⋅).

Condition II: Commutativity
Given dataset , the modes
andℓℓsatisfy commutativity ifℓℓℓℓℓℓ∗

Why LLFC Emerges?
Theorem (Condition I and II imply LLFC)
Given dataset , if two modes
andsatisfy weak additivity for ReLU activations andcommutativity, thenℓℓℓ∗

Weak additivity for ReLU activations and commutativity are verified empirical for modes
that satisfy LMC/LLFC.

Justification of Permutation Method
Given a modeis satisfied:that satisfy LLFC, the commutativity

and a permuted modeℓℓℓℓℓℓℓ

Rewritten as:
ℓℓ(ℓ)

ℓℓℓℓ

Connection to permutation methods
weight matching: min

𝑾

(ℓ)

(ℓ)

−𝑷 ℓ 𝑾

𝑷ℓℓ

Activation matching: min

𝑯ℓ

(ℓ)

(ℓ)

−𝑷 ℓ 𝑯

The two objectives correspond to the two factors of above equation.

ℓConclusionConclusion• Identify Layerwise Linear Feature Connectivity (LLFC)• Investigate the underlying contributing factors to LLFC
• Obtain novel insights into permutation methods
Future Directions• Feature averaging methods• Find a permutation directly enforcing the commutativity property• Going Beyond Neural Network Feature Similarity: The Network FeatureComplexity and Its Interpretation Using Category Theory