Understanding Mode Connectivity via Parameter Space Symmetry

Authors: Bo Zhao, Nima Dehmamy, Robin Walters, Rose Yu
Primary area: learning theory
Keywords: symmetry, mode connectivity

Abstract:
Neural network minima have been observed to be connected by curves along which train and test loss remain nearly constant, a phenomenon known as mode connectivity. 
While this has enabled applications such as model merging and fine-tuning, its theoretical explanation remains unclear. 
We propose a new approach to exploring the connectedness of minima using parameter space symmetry.
By linking the topology of symmetry groups to that of the minima, we derive the number of connected components of the minima of linear networks and show that skip connections reduce this number. 
We then examine when mode connectivity and linear mode connectivity hold or fail, using parameter symmetries which account for a significant part of the minimum.
Finally, we provide explicit expressions for connecting curves in the minima induced by symmetry. 
Using the curvature of these curves, we derive conditions under which linear mode connectivity approximately holds.
Our analysis highlights the role of continuous symmetries in understanding the neural network loss landscape.

Decision: reject
Reviewer scores: 8, 6, 3, 6, 8

Reviews and discussion:
<thread>
creation date:
1730287521712

thread_type:
official_review

<thread_object>
creator: reviewer_anjo

<content>
summary:
This paper introduces a topological framework to understand mode connectivity in general, and linear mode connectivity in particular. Initially, the authors use topological structures to calculate the number of connected components at the 0-level set of the quadratic loss for invertible multi-layer perceptrons. For one-dimensional spaces, they demonstrate that residual connections reduce the number of connected components. Subsequently, they examine the influence of permutations on connectedness and provide a setup where connectedness fails. Finally, conditions are presented for achieving low-loss curves that connect modes.

strengths:
The paper proposes a very powerful framework to analyze properties of the loss surface of deep neural networks. In particular, understanding mode connectivity can shed light on the ways to optimize networks more efficiently. The paper generally provides clear and concise explanations of the proofs for its theorems and propositions.

weaknesses:
However, in its current form, the paper does not fully clarify the connection between topological concepts and the loss landscape of deep neural networks. While it opens with a detailed and precise introduction to topological concepts, it then directly applies these concepts to loss surfaces and networks without discussing necessary assumptions (such as the invertibility of all networks considered). Additionally, there is little exploration of how the topological concept of connected components relates to the depth of the network or which elements correspond to orbits or groups, potentially with examples. Although all this does not detract from the technical accomplishments, it may reduce the paper’s impact by making it difficult for an unprepared reader to link the framework to neural network applications.

In the final section, the concept of curvature is used without clearly defining it in this context. Further, this section could connect more directly to practical applications by demonstrating empirical curves that connect modes and align with derived formulas (e.g., regarding loss growth). A similar issue is in Section 5 concerning symmetries.

Minor Issues:

- one of the first works on the algorithms for finding connectivity is [1], not [2]

- the parameter space (Param) is referenced in Section 3.3 before being formally defined

- please use \citep where appropriate (e.g., the last paragraph of Section 5.2)

[1] Singh, Sidak Pal, and Martin Jaggi. "Model fusion via optimal transport." 

[2] Ainsworth, Samuel K., Jonathan Hayase, and Siddhartha Srinivasa. "Git re-basin: Merging models modulo permutation symmetries."

questions:
1 - The networks analyzed are invertible up to the output layer, meaning the output dimension matches the input dimension. How strictly is this condition required? Does switching to a one-dimensional output immediately yield negative results, as suggested in Section 5.2?

2 - According to [3], layer-wise mode connectivity is achievable. Does Proposition 5.3 contradict this result, or is there a possible connection?

3 - Is proposition about residual connections restricted to 1 dimension?

[3] Adilova, Linara, Asja Fischer, and Martin Jaggi. "Layer-wise linear mode connectivity."
</content>
</thread_object>

</thread>
<thread>
creation date:
1730393155758

thread_type:
official_review

<thread_object>
creator: reviewer_mbky

<content>
summary:
This paper introduces a method to determine the number of components achieving zero loss in linear regression by identifying a homeomorphism between the symmetry in parameter space and the set of parameters yielding zero loss. It demonstrates that permutations can link previously isolated components and offers a novel perspective on the effectiveness of residual connections.

strengths:
1. The concept of employing a homeomorphism to connect the general linear group with the set of parameters yielding zero loss, and quantifying the loss basins by counting the connected components, is both innovative and intriguing.
2. This paper offers a fresh rationale for the effectiveness of residual connections by examining the connected components within the minima of the loss function.
3. In Section 5.1, the paper shows that permutations can link otherwise disconnected loss minima.

weaknesses:
1. Lack of limitations: Since the homeomorphism is specifically tailored to linear regression models, it is necessary to clearly state this limitation in the introduction.
2.

questions:
1. Is it possible to experimentally validate the results of Sec. 6? Can we confirm that the valleys of the loss lines predicted by Eq. 7 correspond to the valleys in the two-dimensional heatmap of the loss landscape?
2. Is it possible to extend the analysis to more realistic models such as ResNet, which has a softmax function, using parameter symmetry and homeomorphic mapping?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730470013309

thread_type:
official_review

<thread_object>
creator: reviewer_srzz

<content>
summary:
This paper studies the space of minima of neural networks (mostly in the linear case) via the angle of group symmetries. A number of standard results in math are listed, and then a number of small results about the structure of minima are given, with some emphasis on the role of skip connections and on the permutations. Each result highlights the role of group actions and this is the main originality of the paper. Overall, I do not see any results that are particularly striking or insightful to understand neural networks in practice. The derivations do not shed a light on any phenomenon of interest, and the presence of group actions when it is not already almost trivial is not established. As a result, while the idea of looking at things from the angle of group action is appealing and elegant, it does not bring convincing additional insight to study the hard question of the minimum landscape of nonlinear neural networks. I would encourage the authors to keep looking in this direction, but something striking would be needed for me to excited.

strengths:
Rigorous, mathematically clean, reasonably clear article.

weaknesses:
The results are not very strong or interesting. Many standard basic results of math are presented like big theorems. Overall, this article does not make things simpler or clearer.

questions:
Is there any reason to believe that there is a natural group action like the ones you explain beyond the cases where it is intuitively clear that there is one? 

Can simulations reveal the presence of group actions or the relevance of your derivations beyond the obvious cases?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730686673363

thread_type:
official_review

<thread_object>
creator: reviewer_mzke

<content>
summary:
The authors investigate mode connectivity from a mathematical perspective. Several results are obtained: (1) The number of connected components of the set of minimizers is characterized in case of linear networks with and without skip connections, where adding skip connections reduced the number of components. (2) In case of 2 layers, they show how permutations can map points to different components, thus “connecting” them and shedding light on recent empirical observations. (3) Next the authors also show that linear mode connectivity does not hold in case of ReLU networks, and that permuting the last two layers does not reduce the barrier either. (4) Finally, the authors characterize non-linear paths connecting such minima and obtain bounds on their curvature, which measures how far away one is from a “linear mode connectivity” regime.

strengths:
1. The paper is well-organized and states all the mathematical results in the Preliminaries section, making this a largely self-contained work and thus easier to read and understand. Linear mode connectivity is still lacking a proper mathematical understanding to this day, making this submission thus a timely contribution.
2. The authors manage to gain quite some insight into the problem with rather mathematically elementary tools, relying on topological properties and results from group theory. I appreciate the result showing that skip connections reduce the number of components, which is in-line with what people observe in practice in terms of easier optimization. 
3. It is also quite nice that in case of two layers, the authors manage to show that permutations indeed connect the components back. While things in practice might be significantly more complicated than the setting considered in this work, I still believe this is a good first step towards obtaining a better understanding of this intriguing phenomenon.

weaknesses:
1. What does it mean intuitively and geometrically if two network parameters are in the same connected component? As the authors point out, this does not imply being path-connected, so while it sounds convincing at first, it is actually not clear to me how this notion ties back to the usual geometric understanding of connectivity used in deep learning. I would appreciate if the authors could clarify things. I.e. how pathological are counter-examples of “connected but not path-connected”?
2. One weakness in the “large barrier” type of results is their global nature, i.e. there is no notion of what types of minima SGD actually finds. The counter-example (as far as I understood) for which a path is constructed for Prop 5.3 and 5.4 starts from a set of parameters and then constructs a new one using the rescaling symmetry. It is not clear to me how “degenerate” these solutions are in the sense that SGD might never choose them due to its implicit bias. I believe there are actually results that show that SGD prefers certain parameters out of the re-scale orbit. In general I think it would be important to better highlight that this work deals with the loss landscape in a global sense, and is not restricted to the minimizers discovered by SGD.
3. The word “connected” has several meanings in this work and I sometimes was confused which one is currently used in a given part of the text. E.g. when two points are in the same connected component (e.g. when mapping with permutations), this is not the same thing as when two points cannot be connected linearly etc. I feel like the manuscript could do a better job at distinguishing these things.

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730722271703

thread_type:
official_review

<thread_object>
creator: reviewer_tnu1

<content>
summary:
The paper studies mode connectivity in neural networks modulo parameter space symmetries from the perspective of topology. 

Authors begin by the counting the connected components of observation that for Euclidean distance as the loss function, the minima of a deep linear network with $l$ hidden layers and invertible weight matrices has $2^{l-1}$ connected components, followed by the observation that having skip connections similar to ResNets reduces the space of connected components. 

For deep linear networks, authors show that one can reduce connected components if one takes into account for permutations. 

Authors then use layer wise scale symmetries in deep networks to show that linear mode connectivity doesn’t hold, however if one controls the weight norms for each layer, we can control the error barrier incurred by linear interpolation within the same connected component. 

Finally, authors introduce general symmetry induce curves that parameterizes the level set of the loss, authors use to curvature of the curve to give sufficient condition for when approximate linear connectivity holds.

strengths:
- This is a well written paper and authors include sufficient background to make the paper readable for a non-expert in topology, like myself to follow all the results and make inferences.
- Authors provide a novel analysis studying mode connectivity in deep linear networks.
- Authors make a number of contributions studying necessary conditions for mode connectivity and approximate linear connectivity that can be used to understand symmetries in neural networks.

weaknesses:
- I see that there is no dependence on width of the network when considering the number of connected components, however permutation symmetries grows exponentially with the width of the network. This appears to be due to the fact we are studying a very simplified setting of linear networks. It’ll be useful for the readers to include a discussion on this.
- It’ll be improve the paper further to improve an example / figure for section 5.1 where permutations leads to mode connectivity.- It is well known that scale symmetries lead to a failure of linear mode connectivity however it is interesting that controlling the weight norms and control over the curvature leads to approximate linear connectivity.
    - How does this relate to  empirical solutions explored by SGD? Specially because it appears that weight decay is necessary for lmc mod permutations.

questions:
It is well known that scale symmetries lead to a failure of linear mode connectivity however it is interesting that controlling the weight norms and control over the curvature leads to approximate linear connectivity.

- How does this relate to  empirical solutions explored by SGD? Specially because it appears that weight decay is necessary for lmc mod permutations.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper studies mode connectivity in neural networks taking a viewpoint from topology, and it provides a number of results on the structure of the loss landscape with particular emphasis on skip connections and permutations, highlighting the role of group actions. Specifically, the number of connected components at the 0-level set of the quadratic loss for invertible multi-layer perceptrons is computed. For one-dimensional spaces, it is shown that residual connections reduce the number of connected components. Next, the impact of permutations on connectedness is examined and an example where connectedness fails is provided. Finally, the authors characterize non-linear paths connecting minima and obtain bounds on their curvature.

All the reviewers agree that the topological perspective on mode connectivity is original, the results are new, and the paper is well written and accessible to a broad audience. These are all strengths of the paper. However, reviewer SRZz has raised an issue concerning the significance of the results. While simple results can in general be impactful, I concur with reviewer SRZz that the usefulness of the topological toolbox developed here towards understanding the loss landscape of neural networks remains unclear. More specifically, the first papers on mode connectivity date back to 2018; since then, there has been a lot of work in this direction, and the advantage of the methodology pursued by the authors w.r.t. this body of research is not evident.

In summary, the paper fails to provide a strong, novel, convincing insight regarding mode connectivity and, for this reason, I recommend rejection.

I still find the perspective taken by the paper interesting and I do recommend that the authors keep working in this direction and provide more convincing evidence of the effectiveness of their approach. One possible direction (mentioned by multiple reviewers) would be algorithmic, in terms of the characterization of the implicit bias of (S)GD.