Understanding Mode Connectivity via Parameter Space Symmetry
Authors: Bo Zhao
Nima Dehmamy
Robin Walters
Rose Yu
Venue: ICML 2025 poster (2025)
Keywords: symmetry
mode connectivity
Primary area: deep_learning
TL;DR: We study why and when mode connectivity holds by linking topological properties of symmetry groups to those of the minimum.

Abstract: Neural network minima are often connected by curves along which train and test loss remain nearly constant, a phenomenon known as mode connectivity. While this property has enabled applications such as model merging and fine-tuning, its theoretical explanation remains unclear. We propose a new approach to exploring the connectedness of minima using parameter space symmetry. By linking the topology of symmetry groups to that of the minima, we derive the number of connected components of the minima of linear networks and show that skip connections reduce this number. We then examine when mode connectivity and linear mode connectivity hold or fail, using parameter symmetries which account for a significant part of the minimum. Finally, we provide explicit expressions for connecting curves in the minima induced by symmetry. Using the curvature of these curves, we derive conditions under which linear mode connectivity approximately holds. Our findings highlight the role of continuous symmetries in understanding the neural network loss landscape.

Understanding Mode Connectivity via Parameter Space Symmetry
Bo Zhao! Nima Dehmamy?
Robin Walters 3 Rose Yu'
Abstract
(Gotmare et al., 2018). Mode connectivity has also inspired applications in diverse fields, including model ensembling (Garipov et al., 2018; Benton et al., 3021; Benzing et al., 2022), model averaging (Izmailov et al., 2018; Wortsman et al., 2022), pruning (Frankle et al., 2020), improving ad- versarial robustness (Zhao et al., 2020), and fine-tuning for altering prediction mechanism (Lubana al., 2023).
Neural network minima are often connected by curves along which train and test loss remain nearly constant, phenomenon known as mode connectivity. While this property has enabled ap- plications such as model merging and fine-tuning, its theoretical explanation remains unclear. We propose new approach to exploring the connect- edness of minima using parameter space symme- try. By linking the topology of symmetry groups to that of the minima, we derive the number of connected components of the minima of linear networks and show that skip connections reduce this number. We then examine when mode con- nectivity and linear mode connectivity hold or fail, using parameter symmetries which account for a significant part of the minimum. Finally, we pro- vide explicit expressions for connecting curves in the minima induced by symmetry. Using the curvature of these curves, we derive conditions under which linear mode connectivity approxi- mately holds. Our findings highlight the role of continuous symmetries in understanding the neu- ral network loss landscape.
Despite extensive empirical validation, mode connectivity, especially linear mode connectivity, remains largely theo- retical conjecture (Altintas et al., 2023). The limited theoret- ical explanation suggests a need for new proof techniques. In this paper, we focus on parameter symmetries, which en- code information about the structure of the parameter space and the minimum. Our work introduces a new approach towards understanding the topology of the minimum and complements existing theories mode connectivity (Yunis et al., 2022; Freeman & Bruna, 2017; Nguyen, 2019; 2021; Kuditipudi et al., 2019; Shevchenko & Mondelli, 2020; Nguyen et al., 2021).
Discrete symmetry is known to related to mode connec- tivity. In particular, the neural network output, and hence the minimum, invariant under neuron permutations (Hecht- Nielsen, 1990). Several algorithms have been developed to find optimal permutations for linear connectivity (Singh & Jaggi, 2020; Ainsworth et al., 2023), and Entezari et al. (2022) conjecture that all minima found by SGD are linearly connected up to permutation. Compared to discrete sym- metry, the role of continuous symmetry, such as positive rescaling in ReLU, in shaping loss landscape remains less well studied.
1. Introduction
Among recent studies on the loss landscape, a particularly in- teresting finding is mode connectivity (Draxler al., 2018; Garipov al., 2018) the observation that distinct min- ima found by stochastic gradient descent (SGD) can be connected by continuous, low-loss paths through the high- dimensional parameter space. Mode connectivity has impor- tant implications for other aspects of deep learning theory, including the lottery ticket hypothesis (Frankle et al., 2020) and the analysis of loss landscapes and training trajectories
We explore the connectedness of minimum through continu- ous symmetries in the parameter space. Continuous symme- try groups with continuous actions define positive dimen- sional connected spaces in the minimum (Zhao et al., 2023). By relating topological properties of symmetry groups to their orbits and the minimum, we show that both continu- ous and discrete symmetry are useful in understanding the origin and failure cases of mode connectivity. Additionally, continuous symmetry defines curves on the minimum (Zhao et al., 2024). This enables a principled method for deriving explicit expressions for paths connecting two minima, a task that previously relied on empirical approaches.
'niiversity of California, San Diego ²IBM Research NNortheastern University. dence Bo Zhao @ucc.euuu....du>>... Nima Dehmamy <nima www.amamhhmmmmmmeee @etiin@@immm..mmmmm. Robin Walters wwwattes@@nrthhasters
Correspon-
Rose Yu <roseyu ucs..
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Our main contributions are:

---

Understanding Mode Connectivity via Parameter Space Symmetry
Providing the number of connected components of full- rank linear regression with and without skip connec- tions, by relating topological properties of symmetry groups to those of minima.
training data). The width requirement is later improved to N + (Nguyen, 2021).
Others prove connectivity under dropout stability. Kudi- tipudi et al. (2019) show that a piece-wise linear path exists between two solutions of ReLU networks, if they are both dropout stable, or both noise stable and sufficiently over- parametrized. Shevchenko & Mondelli (2020) generalize this proof to show that wider neural networks are more con- nected, following the observation that SGD solutions for wider network are more dropout stable. Nguyen et al. (2021) give a new upper bound of the loss barrier between solu- tions using the loss of sparse subnetworks that are optimized, which is a milder condition than dropout stability.
Proving mode connectivity up permutation for linear networks with invertible weights.
Deriving examples where the error barrier on linear interpolation of minima is unbounded.
Deriving explicit low-loss curves that connect minima related by symmetry, and bounding the loss barrier on linear interpolations between minima using the curva- ture of these curves.
A few papers provide theoretical insights into linear mode connectivity using different approaches. Yunis al. (2022) explain linear mode connectivity by finding convex hull defined by SGD trajectory endpoints. Ferbach et al. (2023) use optimal transport theory to prove that wide two-layer neural networks trained with SGD are linearly connected with high probability. (Singh et al., 2024) explain the topog- raphy of the loss landscape that enables or obstructs linear mode connectivity. Zhou et al. (2023) show that the feature maps of each layer are also linearly connected and identify conditions that guarantee linear connectivity. Altintas et al. (2023) analyze effects of architecture, optimization algo- rithm, and dataset on linear mode connectivity empirically.
2. Related Work
Mode connectivity. Garipov et al. (2018) and Draxler et al. (2018) discover empirically that the minima of neu- ral networks are connected by curves on which train and test loss are almost constant. It is then observed that SGD solutions are linearly connected if they are trained from pre-trained weights (Neyshabur et al., 2020) or share short period of training at the beginning (Frankle et al., 2020). Additionally, neuron alignment by permutation improves mode connectivity (Singh & Jaggi, 2020; Tatro et al., 2020). Subsequently, Entezari et al. (2022) conjecture that all min- ima found by SGD are linearly connected up to permutation. Following the conjecture, Ainsworth et al. (2023) develop algorithms that find the optimal alignment for linear mode connectivity, and Jordan et al. (2023) further reduce the bar- rier by rescaling the preactivations of interpolated networks.
We approach the theoretical origin of mode connectivity via continuous symmetries in the parameter space, a connection that has not been previously established. This connection leads to new topological results and explicit expressions of low loss curves. Using these results, we also contribute to the understanding for linear mode connectivity by providing conditions under which it approximately holds.
is worth noting that linear mode connectivity does not always hold outside of computer vision. Language models that are not linearly connected have different generalization strategies (Juneja et al., 2023). Lubana et al. (2023) further show that the lack of linear connectivity indicates that the two models rely on different attributes to make predictions. We derive new theoretical examples of failure cases of linear mode connectivity (Section 5.2).
Symmetry in the loss landscape. Discrete symmetries have inspired a line of work on loss landscapes. Brea et al. (2019) show that permutations of a layer are connected within a loss level set. By analyzing permutation symme- tries, Şimşek al. (2021) characterize the geometry of the global minima manifold for networks and show that adding one neuron to each layer in minimal network connects the permutation equivalent global minima. Continuous sym- metries have also gained attention in optimization (Badri- narayanan et al., 2015; Petzka et al., 2020; Kunin et al., 2021: Zhao et al., 2022). By removing permutation and rescaling symmetries, Pittorino et al. (2022) study the geom- etry of minima in the functional space. Zhao et al. (2023) find set of nonlinear continuous symmetries that partially parametrizes the minimum. Zhao et al. (2024) use symmetry induced curves to approximate the curvature of the mini- mum. Our paper explores a new application of parameter cemmetry-explainiiggg the connectedness of the minimum.
Theory on connectedness of minimum. Several work explores the theoretical explanation of mode connectivity by studying the connectedness of sub-level sets. Freeman & Bruna (2017) show that the minimum is connected for 2-layer linear network without regularization, and for deeper linear networks with L2 regularization. Futhermore, they show that the minimum of two-layer ReLU network is asymptotically connected, that is, there exists path connect- ing any two solutions with bounded error. Nguyen (2019) proves that the sublevel sets are connected in pyramidal networks with piecewise linear activation functions and first hidden layer wider than 2N, where N is the number of
2

---

Understanding Mode Connectivity via Parameter Space Symmetry
3. Preliminaries
number of components of the image f(X) S Y is at most the number of components of X.
In this section, we review mathematical concepts used in the paper and list some useful results on the number of con- nected components of topological spaces. A more detailed version with proofs can be found in Appendix A.
Let X1,..., Xn be topological spaces. The product space is their Cartesian product X₃ x ... X Xn endowed with the product topology. Denote (x( as the set of connected components of a space X. The following proposition pro- vides a way to count the components of product space.
3.1. Connected components
Consider two topological spaces X and Y. A map f: X Y is continuous if for every open subset U C Y, its preim- age (U) is open in X. If X and Y are metric spaces with metrics dx and dy respectively, this is equivalent to the delta-epsilon definition. That is, f is continuous if at every x E X, for any c > 0 there exists 5 > such that dx (z. 3) < 8 implies dr (f))) [()) < e for all y E X.
Proposition 3.5. Consider topological spaces X1, Xn- Then (X1 x X,,) IT:=0 To (T())))).
3.2. Groups
A group is a set G together with composition law, written as juxtaposition, that satisfies associativity, (ab)c (be)) Y a, b, E G, has an identity 1 such that la = al = Y a E G. and for all a E G, there exists an inverse b such that ah = ba = 1. An action of a group G on a set S is a map- : G x + S that satisfies 1 1- = s for all 8 E
A topological space is connected if it cannot be expressed as the union of two disjoint, nonempty, open subsets. A topological space X is path connected if for every p.g E X, there is a continuous map f [0, 1] + X such that f(0) = p and f11 = q. Path connectedness implies connectedness. The converse is not always true (Lee, 2010), but counterex- amples are often specifically constructed and unlikely to be encountered in the context of deep learning. Path connected- ness can therefore help develop intuition for connectedness, for practical purposes.
and (gg') s g (g' (s) for all g.g in G and all s in
S. The orbit of S is the set O(s) = {s' E S gs for some g E G).
A topological group is a group G endowed with topology such that multiplication and inverse are both continuous. A recurring example is the general linear group GI,,, (R), with the subspace topology obtained from Ra* The group GL,, (R) has two connected components, which correspond to matrices with positive and negative determinant.
The following theorem is the main intuition of this paper and will appear frequently in proofs.
Theorem 3.1 (Theorem 4.7 in (Lee, 2010)). Let X,Y be topological spaces and let f X - Y be a continuous map. If X is connected, then f(X) is connected.
The product of groups GI,..,, Gn is group denoted by G1 ... Gn. The set underlying G1 X ... x Gn is the Cartesian product of GI...., G..,,,n.. The group structure is defined by identity (1, 1), inverse (ol (Gt..... 9n) (0r ..... I, and multiplication rule (9),....n)) (n l/h = (91.91..., In.hh
A map I' is a homeomorphism from X to Y if f is bijective and both f and f-1 are continuous. X and Y are homeo- morphic if such a map exists. (connected) component of topological space X is maximal nonempty connected subset of X. The components of X form a partition of X. The next two corollaries of Theorem 3.1 show that con- nectedness and the number of connected components are topological properties. That is, they are preserved under homeomorphisms.
3.3. Connectedness of groups, orbits, and level sets
From Theorem 3.1, continuous maps preserve connected- ness. Through continuous actions, we study the connected- ness of orbits and level sets by relating them to the connect- edness of more familiar objects such as the general linear group. Establishing homeomorphism from the group to the set of minima requires the symmetry group's action to be continuous, transitive, and free. Here we only assume the action to be continuous and try to bound the number of components of the orbits.
Corollary 3.2. Let f X +Y be a homeomorphism from X to Y, and let U C X be a subset of X with the subspace topology. Then U is connected if and only if f(U) C Y is connected.
Corollary 3.3. Let X be a topological space that has N components. Let Y be a topological space homeomorphic to X. Then Y has N components.
As an immediate consequence of Proposition 3.4, an orbit cannot have more components than the group.
Another consequence of Theorem 3.1 is the following upper bound on the number of components of the image of continuous map.
Corollary 3.6. Assume that the action of a group G on S is continuous. Then the number of connected components of orbit O(s) is smaller than or equal to the number of connected components of G. for all s in S.
Proposition 3.4. Let f X >> Y be a continuous map. The
3

---

Understanding Mode Connectivity via Parameter Space Symmetry
Let X be topological space and L X + R a continu- ous function X. A topological group G' is said be symmetry group of L if L(g =) = L(x) for all g E G and E X. In this case, the action can be defined on a level set of I, (c) with a c E R, as G x (c) + T (c). If the minimum of consists of a single orbit, Corollary 3.6 extends to the number of components of the minimum.
WA
Corollary 3.7. Let L be a function with a symmetry group G. If the minimum of L consists of a single G-orbit, then the number of connected components of the minimum is smaller or equal to the number of connected components of G.
30
60
3.0
W2
W2
(a) Linear
(b) Resnet
Figure 1: Minimum of (a) 3-layer linear net Y W₂ W, WX and (b) 3-layer linear net with residual connection ||Y W X + X)12,, where X = 1, Y = 1, and W1. W2, W3 E R.
Generally, symmetry groups do not act transitively on a level set (c) E X. In this case, the connectedness of the orbits does not directly inform the connectedness of the level set. Nevertheless, since the set of orbits partitions the space, we can use the following bound on the number of components of the space.
Proposition 4.1. There is a homeomorphism between L (0) and (GL,)
Proposition 3.8. Let X be topological space and lel X II X₁ be a partition of X into disjoint subspaces. Then To (X)) S > [x)))
Since (GL,)²-1 has 2 connected components and home- omorphisms preserve topological properties, I.~(()) also has 2 connected components. Note that this number is independent of the network width, due to the fact that GT (R)) has two connected components regardless of n.
Consider a topological space X and a group G that acts on X. Let O = {O....., O,,} be the set of orbits. By Proposition 3.8, the number of components of the orbits give the following upper bound on the number of components of the space: To (X) s > (O:)).
Corollary 4.2. The minimum of L has 2 connected com- ponents.
4. Connected Components of the Minimum
4.2. ResNet with ID weights
In this section, we relate topological properties of symmetry groups to topological propertics of the minimum. In particu- lar, we provide the number of connected components of the minimum when all symmetries are known. Omitted proofs can be found in Appendix C.
The topological properties of the minimum set depend on the architecture. As an example of this dependency, we show that adding skip connection changes the number of connected components of the minimum.
Consider a residual network W2 + +X) and loss function
4.1. Linear network with invertible weights
Let Param be the space of parameters. Consider the multi- layer loss function L: Param - R
L(W3, W2, W1) |Y WW(WWW 1 +eX) (2)
where (W1, W2, W3) E Param = R x R x Raxx, E IR, and data X E BTx" Y E Rmax The following proposition states that for a three-layer residual network with weight matrices of dimension 1 x 1, the number of components of the minimum smaller than that of a linear network without the skip connection.
L: Param + R, (Wi, Wi) + ||Y Wi..W
(1)
where X,Y E RAx are the input and output of the network. In this subsection, we assume that both X, Y have rank h, and Param (Raxky.. Then L is invariant to GL (R)'11 which acts Param by g (WI,.., Wi) for
Proposition 4.3. Let n 1. Assume that X,Y # O. When 0, the minimum of L has 4 connected components. When ε + 0, the minimum of L has 3 connected components.
(g1 92 ..... -t...1-11 1 19,
11
(g1 (91. 1) E (R)
Let I, (c) = (0 E Param: [(0)- = c) be a level set of L. Since || 112 2 0 and L (0) * 0. the minimum value
The = O case follows from Corollary 4.2. For the e + O case, the proof decomposes the minimum of L into two sets S, and So, corresponding to the minima without the skip connection and an extra set of solutions because of the skip connection. Sj is homeomorphic to GL2 G.,, and has
of is O. By relating the topology of GL,, (R) and L (0), we have the following observations on the structure of the minimum of L..

---

Understanding Mode Connectivity via Parameter Space Symmetry
4 connected components. So is line and has I connected component. Two components of Si are connected to So. while the other two components of S are not. Therefore, So connects two components of St. As result, the minimum of I. has 3 connected components.
are connected in L (-)
When the hidden dimension h 2 2, there exists a permuta- tion g such that det(g) 0, and a permutation g such that (g) < O. Therefore, Lemma 5.1 implies the following result that all points on the minimum of L are connected up to permutation.
Figure visualizes the minimum without and with the skip connection. This result reveals the effect of skip connection on the connectedness of the set of minima, which may lead to new explanation of the effectiveness of ResNets (He et al., 2016) and DenseNets (Huang et al., 2017). We leave the connection between the topology of the minimum and the optimization and generalization properties of neural networks to future work.
Proposition 5.2. Assume that h 2 2. For all (Wi,.., Wi), (Wi,..., W() E I--(0), there ex- ists a list of permutation matrices Pi,... Pt-1 such that (W1 Pt, PT Pt-2W1-1 Pt-1, Pi-1Wi) and (W1, W() are connected in L-
The results above are examples where a larger part of the minimum becomes connected after a permutation. More generally, permutation improves mode connectivity in cases where an orbit is not connected due to the symmetry group comprising multiple connected components, the orbit does not reside the same connected component of the mini- mum, and there exists permutation that takes a point on one connected component of the group to another.
5. Mode Connectivity
The previous section counts the connected components of the minimum and shows that the connectedness of the mini- mum is related to the symmetry of the loss function under certain conditions. In this section, we use this insight to explain recent empirical observations that with high proba- bility two points in the minimum are connected, i.e. there is a large connected component. Proofs of this section appears in Appendix D.
5.2. Failure case of linear mode connectivity
As application of obtaining new minima from old ones us- ing symmetries, we show that linear mode connectivity fails to hold in multi-layer regressions. The following proposition says that in neural networks with a homogeneous activation (such as leaky ReLU) between the last two layers, the error barrier in the linear interpolation between two solutions can be arbitrarily large.
Mode connectivity refers to the phenomenon that there exist high accuracy or low loss paths between two minima found by stochastic gradient descent (Garipov et al., 2018). Linear mode connectivity occurs when all points on the linear inter- polation between two minima have low loss values. More recently, permutation of neurons is usually performed to align the two minima before evaluating linear mode con- nectivity (Entezari et al., 2022; Ainsworth et al., 2023). We use the term mode connectivity when we consider arbitrary curves and will specify linear mode connectivity when only linear interpolation is considered.
Proposition 5.3. Consider loss fanction of the following form
L: Param + R, W = (W1,..,, Wi) +
1|Y - Woo Wi-3..., ....... W1, 13)
where f is a function of Wi-2, Wi-3..., W1, and a(cz) = (Z(2)) for all c E R and some k. > O. Assume that ||Y||a # 0 and (-) + 0. Also assume that 2 2. For any positive number b O, there exist W, W E I (0) that belong to the same connected component of (0) and 0 < < 1, such that L ((1 - a)W + +WW) > b.
5.1. Mode connectivity up to permutation
For the family of linear neural networks defined in Section 4.1, we show that permutations allow us to connect points in the minimum that are not connected without permuta- tion. Our results support the empirical observation that neu- ron aligmment by permutation improves mode connectivity (Tatro et al., 2020).
The proof constructs a new point on the minimum from an existing one using the rescaling symmetry of homogeneous functions. The two points can be far apart since the orbit of this group action is unbounded. To provide intuition, Figure visualizes the two points on the minimum of a two-layer network with weights of dimension 1 x 1 and the linear interpolation between them. The linear network used is a special case of homogencous network. Note that our result here does not contradict with the wise connectivity result in (Adilova et al., 2024), as more than one layer of the two minima are different.
Consider again the linear network (1) with invertible weights. When 1 = 2. the minimum of L has two con- nected components corresponding to the two connected components of the GL group. Any g E GL that is not on the identity component can take a point component of the minimum to the other.
connected
Lemma 5.1. Consider two points (W1, W2). (Wi, W2) E
L (0) that are not connected in L L (0). For any g E
GT((h) such that det (g) < 0, g (W1, W2) and (W), 11 W?)
5

---

Understanding Mode Connectivity via Parameter Space Symmetry
other activations with known symmetries, similar results may be derived using the large set of minimum obtained from the group action. Whether the loss barrier the linear interpolation is bounded can depend on the compactness of the symmetry group and the curvature of the minimum. We leave a systematic investigation of the condition for linear mode connectivity to future work.
(W5. W3
akW,, Ws) + aa W;
W2
0
(W W
One possible reason why linear mode connectivity ob- served in practice despite Proposition 5.4 is that only small part of the minima is reachable by stochastic gradient descent due to implicit bias (Min et al., 2021), as other opti- mizers have been observed to find less connected minima (Altintas et al., 2023).
11
mi nimum
W1
Figure 2: Interpolation between 2 minima of loss fune- tion L(W,, W2) = ||Y - W2 222 with I dimensional weights. Loss on the interpolation can be unbounded.
5.3. Linear mode connectivity of orbits
Symmetry accounts for a large part of the set of minima. In particular, given a known minimum æ, the orbit of a defines set points that are also minima. Although not all minima are on the same orbit of known symmetries, each orbit often contains nontrivial set of minima. In this section, examine the error barrier of linear interpolations of minima restricted to an orbit of parameter symmetries.
The loss function considered in Proposition 5.3 is signif- icantly more general than those in Section 5.1. For the architecture, we only require the presence of rescaling symmetry in the last two layers, and f can be any neu- ral network with any activation. Other assumptions of the proposition are also not excessively restrictive, as the labels Y are rarely all zero, and there usually exists a minimum in common machine learning tasks.
When the architecture contains a multiplication of two weight matrices W2W,, where W2 E mkmx W1 RAx, there is a GT, symmetry that acts on (W1, W2) by g (W1, W2) = (gW1, 12g for g E GL.. The following proposition states that point on the linear interpolation of two points in the same orbit can be far away from the orbit. Proposition 5.5. Let A E Baxn be an invertible matrix. Let ser S = {(W1, W2) W1, W2 E Raax W, W,W, = A}- For any positive number > there exist W W" E S and O < < 1, such that minwes ((1-a)W' + aW")-
Proposition 5.3 extends to cases where we allow certain permutations. The following proposition states that under additional assumptions, the error barrier in the linear inter- polation is unbounded even with neuron permutations. The proof construction is similar to that of Proposition 5.3.
Let S, be the set of n X n permutation matrices, where n is the number of columns Wi.
W > b.
Proposition 5.4. Consider the loss function with the same set of assumptions in Proposition 5.3. Assume additionally that there does not exist a permutation P such that every column of Po PW (W; Wi-3..., W1. X)) is in the null space of Wi. For any positive number b > 0, there exist (W1,.., Wi), (W1,... ..., Wi) E L (0) and 0 < a < 1, such that (WI...,, 22-2 = (W1, 2/ and
The structure in the form of W1 W2 is not uncommon in deep learning architectures. Notably, the parameter matrices for queries and keys in the attention function are multiplied directly in this manner (Vaswani et al., 2017), thus admitting the GL, symmetry and having orbits with properties given by Proposition 5.5.
min L((1 - (W Wi) PES.
While the error barrier in the linear interpolation of two minima can be unbounded (Proposition 5.3), this typically occurs when the parameters are allowed to be arbitrarily large. Constraining the parameters to remain bounded en- sures that the loss barrier is bounded above. The following proposition makes this intuition precise for the set of minima consisting of a particular orbit.
+ +a(W,,... Wi-,, P- W1 WiP)) >
By including permutation, the setting in Proposition 5.4 is closer the setting in which linear mode connectivity is empirically observed. However, the permutation in Propo- sition 5.4 is restricted to the first two layers. which does not rule out the possibility of lowering the loss barrier by including permutations of other neurons.
Proposition 5.6. Consider the loss function with the same set of assumptions in Proposition 5.3. Let W E (0) be a point on the minimum. Consider the multiplicative group of positive real numbers I. that acts on L L by g. g(Wi, Wi) = (Wi..., W1-2 WW-1, Wig where
The proofs of Proposition 5.3 and 5.4 depend on the rescal- ing symmetry of homogenenous activation functions. For
6

---

Understanding Mode Connectivity via Parameter Space Symmetry
g E RR. Then there exists a positive number b > 0, such that for all O < < and W' E (r) with [|W]]l2 < for all's i and some > 0, the loss value for points the linear interpolation L ((1 a)W + +aW)== < b.
(6) does not preserve loss when batch size k is larger than hidden dimension h. However, the maximum change of loss y can be bounded as follows.
Proposition 6.1. Let (U.V) E Param, and (U",V" g (U,V). Then
Proposition 5.5 and 5.6 are two examples where the knowl- edge of parameter symmetry enables analysis of the linear connectivity of subsets of minima. As more continuous symmetries are characterized (e.g. the nonlinear symme- tries in Zhao et al. (2023)), these analysis can potentially be extended to even larger parts of the set of minima.
- ((lvx))U''(VV'A) & [VX
(7)
We demonstrate Proposition 6.1 empirically using a set of two-layer networks with various parameter space dimen- sions. Specifically, we construct networks in the form of [Uo((V) - Y112, with a being the sigmoid function, X E Ra*k Y Ra and (U, E Param = 1 x ggxn We create 100 such networks, each with m. h, n, k randomly sampled from integers between 2 and 100. In each network, elements in X and Y are sampled independently from a normal distribution, and U, V are randomly initialized. Af- ter training with SGD, we compute (U", V') = 9 (U, V) using (6) with a random invertible matrix g. We then plot (VX(X))I against (VXx(VX) - U (V'X)l in Figure 3(a). All points are above the line y = x, as predicted by Proposition 6.1.
6. Curves on Minimum from Group Actions
The paths connecting two points in the set of minima may not be linear. Previously, these paths were discovered empir- ically by finding parametric curves which the expected loss is minimized (Garipov et al., 2018). Using parameter space symmetry, we uncover an alternative and principled way to find curves on the minimum.
6.1. Symmetry induced curves
Suppose the loss function L Param + R is invariant with respect to some Lie group G. Consider the following curve for a point w E Param and M E Lie(
While the map (6) is not group action in general, it con- nects more points in the set of minima than only using known symmetries, and the points on the connecting curves have bounded loss. Figure 3(b-c) shows that the loss on the curves induced by approximate symmetries remains rela- tively low, compared to the loss on the linear interpolation between the two ends of these curves. We consider a two layer network with loss function || (W, () X)-Y|l, with being leaky ReLU function, X E IR 16x8 Y E R24x8, and (W1, W2) E Param = 322216 x 32 In the figures, 7 de- notes a curve obtained using Equation (5) together with (6). The starting point of y is a minimum found by SGD. Both 7 and the linear interpolation are parametrized by E [0, 1]. Compared to the linear interpolation between the two end points of 7, the loss on y is consistently lower. Figure 3(c) uses group elements with larger magnitudes, resulting in a larger distance between (()) and (1), which might explain the higher loss barrier on their linear interpolation.
YM IR& x Param + Param,
(t, w) = exp (tM) . w.
(4)
Since exp (tM) E G and the action of G preserves the value of L. every point on Yar is in the same L level set as w. This provides a way to find a curve of constant loss between two points that are in the same orbit. Concretely, given two points wi and W2 wn let.y be the following curve:
7: [0, x G x Param + Param, ((I,g,w) w) exp (llo(g)) w.
(5)
Note that 700 g: w1) = w1, > g. wi) L(7(1, g, w))) L(w; L(w²) for all / E [0, 1]. Hence, is a curve that connects the points wT and u2, and every point on 7 has the same loss value as L(w;)= = L(w2).
w2, and
6.2. Approximate linear connectivity under bounded curvature of minima
For a group G, the curve Y is defined when the map G x Param -> Param is continuous and id e = ur for all w E Param, even if is not a group action or does not preserve loss. However, when does not preserve loss, the loss can change on 7. Consider our two-layer network and the following map:
Knowing the explicit expression of connecting curves brings new insight into when linear mode connectivity approxi- mately holds. In particular, these expressions provide infor- mation about the curvature of the curves. If the curvatures are small, then there exists approximately straight line connecting any two minima along which the loss remains close to its minimum value.
:: GGL(h, IR) Param > Param
g VU, = (Uo(VX)o(gVXX)
When a is the identity function, preserves the loss value, and 7 defines a curve on the minimum. In general, the map
Consider a loss level set (c) = {w E Param I(ww) = c) with some e E R. Suppose we have two points
7

---

Understanding Mode Connectivity via Parameter Space Symmetry
(a)
(b)
(c)
7. Discussion
1ot0
320 0.
230 0.4-
0.8
In this work, we study topological properties of the loss level sets by relating their topology to the topology of sym- metry groups. Specifically, we derive the number of con- nected components of full-rank multi-layer networks with and without skip connections, and prove mode connectivity up to permutation for full-rank linear regressions. Using symmetry in the parameter space, we construct an explicit expression for curves that connect two points in the same orbit. The explicit expressions allow us to obtain the cur- vature of these curves, which are useful to bound the loss barrier on linear interpolation between minima.
220
s00
sco
02.
0b 06
[UotV26
-V'otV201
Figure 3: (a) Empirical validation of Proposition 6.1. (b-e) The loss on the curves induced by approximate symmetries (r) remains relatively low, compared to the loss on the linear interpolation between the two ends of these curves. (b) and (c) differ by the magnitude of the group element used. The loss is averaged over 5 random curves.
For practitioners, these results motivate concrete strategies and cautions-for tasks that navigate the loss landscape, including model merging, ensembling, and fine-tuning:
w1, W2 E (c) connected by smooth curve 7 lying entirely within (c). The curvature of y can be written as t(),t) = (+)l where >' = zे and (t) = Trioll If the curvature of this curve is small or bounded, we can show that there exists an approximately straight line connecting w1 and Wy that remains close to L (c). Additionally, if L Lipschitz continuous, its value remains close to e along this line segment. We formalize this with the following theorem.
7'10)
One can build low-loss curves explicitly using known parameter symmetries. This gives principled and efficient way to obtain new minima from old ones, po- tentially useful for (1) generating model ensembles with low cost; (2) improving model alignment by al- lowing a much larger group of transformations than permutation; and (3) mitigating catastrophic forgetting fine-tuning by constraining updates to the symmetry- induced manifold of the pretraining minimum.
Theorem 6.2. Let (c) C. Param, with E IR, be level set of the loss function L Param + R. Let : [0. + I. (c) be a smooth curve in I, (c) connecting two points ww = (0) and W2 = (1). Suppose the curvature x<(f) of y satisfies x(t)= S Kmmax for all t E [0. 1].
The connectedness of minima supports the practice of model merging and ensembling, even when models are trained separately. In addition permutation, many other symmetry transformations can connect solutions that would otherwise appear very different.
Let S be the straight line segment connecting wi and w2- Then, for any point w on S. the distance to L (c) is bounded by
dist(w, L (c)) S dmax.
(8)
Linear interpolation between minima is not guaranteed to lead to better models, despite its widespread use. This highlights the need to evaluate whether the min- ima found by specific learning algorithms are approxi- mately connected before averaging models directly.
with
Nimax |w2- wi 1|2
dmax
Kiaxx
Furthermore, assuming I, is Lipschitz continuous with Lips- chitz constant Ct, the loss at any point w on S satisfies
Extending these results to nonlinear networks is a challeng- ing yet exciting future direction. A full characterization of the minima in non-linear settings requires identifying the complete set of symmetries an open problem for many architectures-ann and analyzing how the resulting orbits inter- sect, which becomes increasingly complex as the number of orbits grows. One approach is through approximate symme- tries, such as those in Section 6. Another by continuously deforming the network or its minimum and studying its be- havior in the limit as the network approaches linear regime. Additionally, many modern architectures retain large con- tinuous symmetry groups, particularly in components like self attention or normalization layers. As we saw in Section 5.3, even partial knowledge of symmetries in a network can yield valuable structural information about its minima.
[L(w)- -c Cidmax.
(9)
When the group action induces curves with bounded cur- vature, Theorem 6.2 applies. Since the minimum is also a level set of L, Theorem 6.2 provides a sufficient condi- tion for linear mode connectivity to approximately hold. When the curvature of the minimum is small, points on the minimum are approximately connected through nearly straight paths along with the loss does not increase sig- nificantly. If Kmax|| - wi || is small, we can use the first-order approximation of the square root and obtain d Mmuax [ will w
8

---

Understanding Mode Connectivity via Parameter Space Symmetry
Acknowledgements
tion saddles, connected by equal-loss valleys across the loss landscape. arXiv preprint (9X..0:907.029//11 2019.
We thank lordan Ganev for helpful comments on proofs in Appendix A. This work was supported in part by the U.S. Army Research Office under Army-ECASE award W91 1INF-07---0003-03, the U.S. Department Of Energy. Office of Science, IARPA HAYSTAC Program, and NSF Grants #2205093, #2146343, #2134274, #2107256, #2134178, CDC-REF--TT223-0069, DARPA AIE FoundSci and DARPA YFA.
Chazal, F. and Michel, B. An introduction topologiaa data analysis: fundamental and practical aspects for data scientists. Frontiers in artificial intelligence, 4:667963, 2021.
Draxler, F., Veschgini, K., Salmhofer, M., and Hamprecht, F. Essentially no barriers in neural network energy land- scape. In International conference on machine learning, pp. 1309-1318. PMLR, 2018
Impact Statement
This work advances the theoretical understanding of mode connectivity in neural network loss landscapes through pa- rameter space symmetries, with potential applications in model merging and fine-tuning. We do not identify specific ethical concerns but recognize that implications may vary depending on the domain of application.
Entezari, R., Sedghi, H., Saukh, O., and Neyshabur, B. The role of permutation invariance in linear mode connec- tivity of neural networks. International Conference on Learning Representations, 2022.
Ferbach, D., Goujaud, B., Gidel, G., and Dieuleveut, A. Proving linear mode connectivity of neural networks via optimal transport. arXiv preprint urXiv:2310.19103. 2023.
References
Adilova, L., Andriushchenko, M., Kamp, M., Fischer, A., and Jaggi, M. Layer-wise linear mode connectivity. In The Twelfth International Conference Learning Repre- sentations, 2024.
Frankle, J., Dziugaite, G. K., Roy, D., and Carbin, M. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp. 3259- 3269. PMLR, 2020.
Ainsworth, S. K., Hayase, J., and Srinivasa, S. Git re- basin: Merging models modulo permutation symmetries. International Conference Learning Representations, 2023.
Freeman, C. D. and Bruna, J. Topology and geometry of half-rectified network optimization. In Sth International Conference on Learning Representations, ICLR, 2017.
Altintas, G. S., Bachmann, G., Noci, L., and Hofmann, T. Disentangling linear mode-connectivity. arXiv preprint arXiv:2312.09882. 2023.
Gabrielsson, R. B. and Carlsson, G. Exposition and interpre- tation of the topology of neural networks. In 2019 18th ieee international conference on machine learning and applications (iemla), pp. 1069-1076. IEEE, 2019.
Badrinarayanan, V., Mishra, B., and Cipolla, R. Symmetry- invariant optimization in deep networks. arXiv preprint arXiv:15/1.0175: 2015.
Garipov, T., Izmailov, P., Podoprikhin, D., Vetrov, D. P., and Wilson, Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances neural information processing systems, 31, 2018
Benton, G., Maddox, W., Lotfi, S.,, and Wilson, A. G. Loss surface simplexes for mode connecting volumes and fast ensembling. In International Conference on Machine Learning, pp. 769-779. PMLR, 2021.
Gotmare, A., Keskar, N. S., Xiong, C.,, and Socher, R. Us- ing mode connectivity for loss landscape analysis. 35th International Conference Machine Learning's Work- shop Modern Trends in Nonconvex Optimization for Machine Learning. 2018.
Benzing, F., Schug. S., Meier, R., Von Oswald, J., Akram, Y., Zucchet, N., Aitchison, L., and Steger, A. Random initialisations performing above chance and how to find them. 14th Annual Workshop Optimization for Ma- chine Learning (OPT2022), 2022.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Bharadwaj. A. and Hoşten, S. Complex critical points of deep linear neural networks. arXiv preprint arXiv:2301. 12651, 2023.
Bicht-Nielsen, R. On the algebraic structure of feedforward network weight spaces. In Advanced Neural Computers, pp. 135, Elsevier, 1990.
Brea, J., Simsek, B., Illing, B., and Gerstner, W. Weight- space symmetry in deep networks gives rise to permuta-

---

Understanding Mode Connectivity via Parameter Space Symmetry
Huang, G., Liu, Z., Van Der Maaten, L.., and Weinberger, K. Q- Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 47004470000 2017.
Nguyen, Q. On connected sublevel sets in deep learning. In International conference on machine learning, pp. 4790- 4799. PMLR, 2019.
Nguyen, Q. A note on connectivity of sublevel sets in deep learning. arXiv preprint arXiv:2101.08576, 2021.
Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson, A. G. Averaging weights leads to wider optima and better generalization. Conference on Uncertainty in Artificial Intelligence, 2018.
Nguyen, Q. N., Bréchet, P., and Mondelli, M. When are so-
lutions connected deep networks? Advances in Neural Information Processing Systems, 34:20956-20669, 2021.
Jordan, K., Sedghi, H., Saukh, O., Entezari, R., and Neyshabur, B. Repair: Renormalizing permuted acti- vations for interpolation repair. International Conference on Learning Representations. 2023.
Petzka, H., Trimmel, M., and Sminchisescu, C. Notes on the symmetries of 2-layer relu-networks. In Proceedings of the northern lights deep learning workshop, volume 1, pp. 6-6, 2020.
Juneja. J., Bansal, R., Cho, K., Sedoc, J., and Saphra, N. Lin- ear connectivity reveals generalization strategies. Inter- national Conference on Learning Representations, 2023.
Pittorino, F., Ferraro, A., Perugini, G., Feinauer, C., Bal- dassi, C., and Zecchina, R. Deep networks on toroids: Removing symmetries reveals the structure of flat re- gions in the landscape geometry. In Proceedings of the 39th International Conference on Machine Learning, pp. 17759-17781, 2022.
Kohn, K., Merkh, T., Montúfar, G., and Trager, M. Geom- etry of linear convolutional networks. SIAM Journal on Applied Algebra and Geometry, 6(3):368-40))) 4066, 2022.
Shevchenko, A. and Mondelli, M. Landscape connec- tivity and dropout stability of sgd solutions for over- parameterized neural networks. International Con- ference on Machine Learning. pp. 8773-8784. PMLR, 2020.
Kuditipudi, R., Wang, X., Lee, H., Zhang, Y., Li, Z., Hu, W., Ge. R., and Arora, S. Explaining landscape connectivity of low-cost solutions for multilayer nets. Advances in neural information processing systems, 32, 2019.
Kunin, D., Sagastuy-Brena, J., Ganguli, S., Yamins, D. L., and Tanaka, H. Neural mechanics: Symmetry and bro- ken conservation laws in deep learning dynamics. In International Conference on Learning Representations, 2021.
Şimşek, B., Ged, F., Jacot, A., Spadaro, F., Hongler, C., Gerstner, W. and Brea, J. Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances. In International Conference Machine Learning, pp. 9722-97727 27329.732222 PMLR, 2021.
Lee, J. Introduction to topological manifolds, volume 202. Springer Science Business Media, 2010.
Singh, S. P. and Jaggi, M. Model fusion via optimal trans- port. Advances in Neural Information Processing Systems, 33:22045-22055 2020.
Lubana, E. S., Bigelow, E. J., Dick, R. P., Krueger, D., and Tanaka, H. Mechanistic mode connectivity. Inter- national Conference on Machine Learning, pp. 22965- 23004. PMLR, 2023.
Singh, S. P., Adilova, L.., Kamp, M., Fischer, A., Schölkopf, B., and Hofmann, Landscaping linear mode connectiv- ity. arXiv preprint arXiv:2406. 16300, 2024.
Tatro, N., Chen, P.-Y,, Das, P., Melnyk, I., Sattigeri, P., and Lai, R. Optimizing mode connectivity via neuron alignment. Advances in Neural Information Processing Systems, 15555000011111 2020.
Mehta, D., Chen, T., Tang, T., and Hauenstein, D. The loss surface of deep linear networks viewed through the alge- braic geometry lens. IEEE Transactions on Pattern Anal- ysis and Machine Inielligence, 44(9):5664-568) 2021.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, E., and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Min, H., Tarmoun, S.. Vidal, R., and Mallada, E. On the plicit role of initialization on the convergence and implicit bias of overparametrized linear networks. In Interna- tional Conference on Machine Learning, pp. 7760-7768. PMLR. 2021.
Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong. H., Farhadi, A., Carmon, Y., Kornblith, S., et al. Model soups: averaging weights of multiple fine-tuned mod- els improves accuracy without increasing inference time.
Neyshabur, B., Sedghi, H., and Zhang, C. What is being transferred in transfer learning? Advances in neural information processing systems, 33:512-523, 2020.
10

---

Understanding Mode Connectivity via Parameter Space Symmetry
In International Conference Machine Learning, pp- 33965-23998 PMLR, 2022.
Yunis, D., Patel, K. K., Savarese, H. P., Vardi, G., Frankle, J., Walter, M., Livescu, K., and Maire, M. On convexity and linear mode connectivity in neural networks. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop), 2022.
Zhao, B., Dehmamy, N., Walters, R., and Yu, R. Symmetry teleportation for accelerated optimization. Advances in Neural Information Processing Systems, 2022.
Zhao, B., Ganev, I., Walters, R., Yu, R., and Dehmamy, N.
Symmetries, flat minima, and the conserved quantities of gradient flow. International Conference on Learning Representations, 2023.
Zhao, B., Gower, R. M., Walters, R., and Yu, R. Improving convergence and generalization using parameter symme- tries. International Conference on Learning Representa- tions, 2024.
Zhao, P., Chen, P.-Y., Das, P., Ramamurthy, K. N., and Lin, X. Bridging mode connectivity loss landscapes and adversarial robustness. International Conference on Learning Representations, 2020.
Zhou, Z., Yang, Y., Yang, X., Yan, J., and Hu, W. Going beyond linear mode connectivity: The layerwise linear feature connectivity. arXiv preprint 23xv:23077.0828. 2023.

---

Understanding Mode Connectivity via Parameter Space Symmetry
Appendix
A. Background
This section contains additional background in general topology and proofs for statements Section 3. We refer readers to (Lee, 2010) for more detailed introduction this topic.
A.1. Connected components
Consider two topological spaces and Y. map f X + Y continuous for every open subset U C Y, its preimage f- (U) is open in X. If X and are metric spaces with metries dx and dy respectively, this equivalent the delta-epsilon definition. That is, f is continuous if every x E X, for any O there exists >> such that dx (ar, y) < 8 implies dy (zz) f())) < for all y X.
A topological space is connected if it cannot be expressed as the union of two disjoint, nonempty, open subsets. A topological
space X is path connected if for every p, E X, there is a continuous map f [0, 1] + X such that f(0) = and f(1) q. Path connectedness implies connectedness, but the converse is not true (Lee, 2010). (Nguyen, 2019) studies the path connectedness of sublevel sets of loss functions.
The following theorem is the main intuition of this paper and will appear frequently in proofs.
Theorem A.1 (Theorem 4.7 (Lee, 2010), Theorem 3.1 in the Main Text). Let X, Y be topological spaces and let f: Y be a continuous map. If X connected, then f(X) connected.
A map f is a homeomorphism from to f is bijective and both f and f are continuous. X and Y are homeomorphic such a map exists. A (connected) component of a topological space X is a maximal nonempty connected subset of X. The components X form a partition of X. The next two corollaries of Theorem A.1 show that connectedness and the number of connected components are topological properties. That is, they are preserved under homeomorphisms.
Corollary A.2. Let f: X + Y be a homeomorphism from X to Y, and let U C X be a subset of X with the subspace topology. Then U is connected if and only if fU) C Y connected.
Proof. By the definition of homeomorphism, f and /-1 are continuous. From Theorem A.1, U E X connected, then f(U) E is connected. Similarly, if fUU connected, then (f(()( U is connected.
Corollary A.3. Let X be a topological space that has N components. Let Y topological space homeomorphic to X. Then Y has N components.
Proof. Let Ciy..., ON be the components X. Let f be a homeomorphism from X to Y. Since f is bijective and Ci...,, CN is a partition of X, f(C)). f(CN) partition of Y. From Theorem A.1, since C1, ..., CN are all connected, so are ((CC),
Lastly, we need show that f(, are maximally connected. Suppose there exists set U C Y, such that U & f(C) and f(C)) U is connected for some i. Then by Theorem A.1. f (f Uf( - C₁ is connected in X. This
contradicts the fact that C, is maximal component in X. Therefore, f(C f(CN) are maximally connected.
Since f(Ci), ((Cx) partitions and are maximally connected, Y has N components.
Another consequence of Theorem A.1 is the following upper bound on the number of components of the image of a continuous map.
Proposition A4. Let: f X + Y be continuous map. The number of components of the image f(X) S Y is at most the number of components of X.
Proof. Let Ci...., ON be the components of X. Since C, is continuous and the action is continuous, according to Theorem A.1, (C,) is continuous for all i E (1,..., Additionally, since U C, = X, we have U f(C,) = =(X). Therefore, there is a surjective map from (((()) to the set of components of f(X), which implies that f(X) has at most N components.
12

---

Understanding Mode Connectivity via Parameter Space Symmetry
Let X1,.., X, be topologcaa spaces. The product space is their Cartesian product X1 x x... x Xn endowed with the product topology. Denote Tx( as the set of connected components of a space X. The following proposition provides way count the components of a product space.
Proposition A.S. Consider n. topological spaces X1,.., Xn. Then (o x ... x Xn)] - I (X,))
Proof. When I, the number of components of the product space is [o(X))].
For the n > 1 case, since X1 x... x X,, = (X1 ... x Xn-1) x X,, it suffices to show that (o((A x B) = (TA)l) for any topological spaces A and B. Let f : To(A) x To( + To (A x B) be the map that assigns C' E TTo ()) x TO (B) to the element in (A x B) that contains C. Then f is surjective because To (A) x To(B) forms a partition of A x B. To prove that o is injective, suppose that f(C1) = f(C2) for C1, C2 E mb(A) x To B). Consider the projection FΛ A x B - A. Since A is continuous and C₁,Cr belong to the same component of x B, a (C1) and TA(C2) belong to the same component of A. Similarly, T T and (C() belong to the same component of B under the projection TB : A B + B. Since all components of A and B are maximally connected, we have C₁ C22 which implies that f is injective. Since bijection from (A) x To B) to To B), (o(A x B)| [m)(A)][
A.2. Groups
A group set G together with composition law, written as juxtaposition, that satisfies associativity, (ab)e a(be) V a, b, E G, has an identity such that 1a = al V a E G, and for all G. there exists inverse b such that ab- = ba = 1. An action of a group G on a set S is map G x S - S that satisfies 1 = for all 8 E S and (gg').s .s = g- 9-' s) for all 9,g6 in G and all s in S. The orbit of s E S is the set O(s) = {s' E S s = gs for some E G}.
A topological group is a group G endowed with a topology such that multiplication and inverse are both continuous. A recurring example is the general linear group GLn, (R), with the subspace topology obtained from R=²,, The group GL,, (R) has two connected components, which correspond to matrices with positive and negative determinant.
The product of groups Gi,... Gn a group denoted by G1 x... x Gn. The set underlying G1 x ... x G, is the Cartesian product of Gi...., Gn,- The group structure is defined by identity (1, 1), inverse (g1. 9n) (g) 9n. 1) and multiplication rule (f1....,hh In) (g': ..... I(h) (g19's -.. 9n9h
A.3. Relating connectedness of groups, orbits, and level sets
From Theorem 3.1, continuous maps preserve connectedness. Through continuous actions, we study the connectedness of orbits and level sets by relating them to the connectedness of more familiar objects such the general linear group. Establishing homeomorphism from the group to the set of minima requires the symmetry group's 's action to be continuous, transitive, and free. Here we only assume the action to be continuous and try to bound the number of components of the orbits.
As an immediate consequence of Proposition A.4, an orbit cannot have more components than the group.
Corollary A.6. Assume that the action of a group G on S is continuous. Then the number of connected components of orbit (s) is smaller than or equal to the number of connected components of G. for all in S.
Proof. An orbit (ss) is the image of the group action, which we assume to be continuous. The result follows from Proposition A.4.
Let X be a topologiaal space and L X + R a continuous function on X. A topological group G is said to be a symmetry group of L L(g 2) = L(x) for all g E G and x E X. In this case, the action can be defindd on a level set of L, (c) with e E RR as G x -() If the minimum of L consists of a single orbit, Corollary A.6 extends immediately to the number of components of the minimum.
Corollary A.7. Let L. be a function with a symmetry group G. If the minimum of L, consists of a single G-orbit, then the number of connected components of the minimum is smaller or equal the number of connected components of G.
Generally, symmetry groups do not act transitively on a level set L L-((c) E X. In this case, the connectedness of the orbits does not directly inform the connectedness of the level set.
13

---

Understanding Mode Connectivity via Parameter Space Symmetry
Proposition A.8.
1. There exists space X and group G with action on X, such that each orbit for the group action is connected and X is not connected.
2. There exists space X and group G with an action on X, such that each orbit for the group action is disconnected and X is connected.
Proof. For part (a), consider subspace of R X X1 U22 where X1 [xx,)) = 0} and X₂ {(x, y) = 1, 0). The space X is not connected. Let G' be the multiplicative group of positive real numbers and act on X by multiplication on the second coordinate. Then there are two orbits, X1 and X2, which are both connected.
For part (b), consider the space X = R2 {0}. Then is connected. Let be the multiplicative group of real numbers, which acts on X by multiplication on both coordinates. That is, g (21, x2) = (gx, gr2), v(11, E X, Vg E G. The orbit of any point (I1, x2) E X is not connected.
Nevertheless, since the set of orbits partitions the space, we can use the following bound on the number of components of the space.
Proposition A.9. Let X be a topological space and let X = [I X; be a partition of X into disjoint subspaces. Then (oo (X)) < Σ, mo(())))))
Proof. Let = {AC A is a component of } be the union of the components of the subspaces. Then S is a partition of X, and every element in S is connected. Therefore, there surjective map from S to To(X), defined by mapping each s E S to the element of (X)) that includes s. This implies that (xo( S |S| Em= (X,().
Consider a topological space X and group G that acts on X. Let O {O1,.... A.9, the number of components of the orbits give the following upper bound on the number of components of the space: |ma(X) < Σ (O,))-
.... O₂} be the of orbits. By Proposition
B. Additional Related Work
Topological approaches in machine learning. Topology has been applied in other areas of machine learning, particularly through tools such as persistent homology, to study the structure of data manifolds and training dynamics (Chazal Michel, 2021). For example, prior work has used topological data analysis (TDA) study the shape of activation patterns, understand generalization, and visualize learning trajectories (Gabrielsson & Carlsson, 2019).
Bounds the number of connected components. For neural networks that are systems of polynomials, the number of critical points can be upper bounded using methods in algebraic geometry. Mehta et al. (2021) shows that after adding generalized L2 regularization, there are no positive- positive-dimensioe dimensional solutions in deep linear networks with mean squared error. They observe that the critical points, which satisfy V = O, form the solution set of a system of polynomial equations. They then provide two upper bounds, the classical Bezout bound (CBB) and the (BKK), on the number of isolated complex critical points. Bharadwaj & Hoşten (2023) improves the upper bound for neural networks with one hidden layer and one training data point. Kohn al. (2022) provide bounds on the number of critical points in the function space for linear convolutional networks. Since proving the exact number of connected components of minimum is not always easy, a possible future direction is to derive Bezout and BKK bounds the number of connected components for various architectures and perhaps extend this analysis beyond polynomial neural networks.
Bound
C. Proofs in Section 4
Proposition 4.1. There is homeomorphism between I, (0- and (GL,,) 1-1
14

---

Understanding Mode Connectivity via Parameter Space Symmetry
Proof. Recall that W1, W.... Wn, X, Y matrices in Rxxx, and X, Y are both full rank. Consider the map
1-1 II g.
f: (GLAA 1 (0), -01...99--11 + (g) 1- 92. ... gi 1, Y
(10)
The inverse /- : (Wi,.., Wi) + (WiX, W2, Ws.,., W(-1) is well defined, because X, WT, W2, W3, Wi-1 are all full-rank. Since both f and f-1 are continuous, f homeomorphism between (GL,))=1 and L (0).
Corollary 4.2. The minimum of L has 2 connected components.
Proof. From Proposition 4.1, (0) homeomorphic (GLA)--1 According to Corollary A.3, this implies that (0) has the same number of connected components as (GL,))* From Proposition A.5, GL, (R) (R)--1 has 2 connected components. Therefore, L (0) has 2' connected components.
Proposition 4.3. Let = 1. Assume that X,Y + O. When 0, the minimum of L has 4 connected components. When E + O, the minimum of L has 3 connected components.
Proof. When E = O, the skip connection is effectively removed, and the loss function (2) reduces to (1). By Corollary 4.2, the minimum of I. has 4 connected components. In the rest of the proof, we consider the case where 6 # 0.
Let (W lo: W2., W3,) = (1, (a - =)4, a - where a E is an arbitrary number such that a * E and a A 0. Then (W 1.: W2. W3,) is point in (O). Define set G1 {9 E Rhxh det (g W1, + +X) # 0}. Let
a: GTa x G1 Param be the following map:
91.92 (g) W
92 W2.9.
Wa ( 2W2, Wi, X + (2x) Wi. X + +
(11)
From the definition of G1, 22, W1, X + =X) is invertible, so a is well defined. Additionally, we have L(a(g, 92))
=
L(W1.: W2., Ws,) 0, E GL x G1. Therefore, denoting the image of a as S1, we have S1 C L L-)
Let So {(W,, 22 W3) : W3 ((x)) and W1 0} if # or 0 otherwise. For (W1, W2, Wa) E Sp. we have L(W,, W2, W3) |Y - ((xX) (0+ eX|1 = 0. Therefore, So L-
We then show that the minimum of 7. is the union of S and Sp. Consider point (W1, W2, W3) E (.) If = 0. then * 0, otherwise (W1, W2, W)) cannot be in --) In this case, W3 must equal to (xx)) and (W,, W2, W/)) E S. If W, + 0, then W WT.1 E GLI and W2W, 1o -W E G1. The second part is due to W
Wi, = W2W,X + * 0 since (W1, W2, W3) E L L-) In this case we have (W1, W2. Wa) a(W, W2 W which means that (W1, W2, Ws) E S1.
The number of connected components of S₁ and S. can be obtained from their structures. Since W2, Wi, X * 0, there is a homeomorphism between 1₁ and GL1 defined by the map
G + GL1, + gW,, W1, +
(12)
with inverse /² :LI + G1,g + slg - xX) x) Since a is also a homeomorphism, its image S is homeomorphic GL, GL1 and has connected components. When * So is a line and thus has connected component.
The last part of the proof shows the connectedness of the connected components of S and Sp. Let G {02 E G f(g2) E (L)) be the connected component in G2 that correspond to GL (i) and GT = {92 E G, f(g)) e GL (i-s be the component that correspond to GL -iggu((())), For convenience, we name the connected components of Im(a) as follows:
G = {W1, W2, W3) E Param (W1, W2, W₂) = a(g1 (01. g1 E GL .22 E G
C2 = {W1, W2, W3) E
Param (W1, W2, W3) = al91..2).90 91 E GL E Gt}
C. {(W1, W2, W3) E
Param (W1, W2, W3) o(91.92): 91 E GL* 92 E GT}
C4 (WW,, W2, Wa) E
Param (W1, W2, W3) o(91,92).9)) E GL .2 E GT}
15

---

Understanding Mode Connectivity via Parameter Space Symmetry
Note that for (W1, 22, W3) S1, there exists (unique) 02 E G1 such that we can write W3
= W3 33. 222 Wi, i eX] WT, - Y
Following from the definition of G for point (W1, 22, Wa) in C₁ or C2, sigg(WW)) = (Y Additionally, when 82 is close to 0, 92 belongs to G The boundary of both C1 and C2 contain point in Sg:
lim (( lim a(9).)) = (0, a - e, ((xX)-1) E Sn- +1+
91-00
Therefore, both Cj and C2 are connected Sp-
For points in C3 and Ca. sis) + sign(Y(ex))))). Therefore, no point in C3 or Ca can be sufficiently close to Sg. As result, these components are not connected to So. In summary, when e o 0, So connects 2 components of S1, and the minimum of L has 3 connected components.
We note that connectedness alone does not imply easy connectivity in the sense of short or simple paths between solutions. Being in the same connected components is necessary condition for connectivity, but single component may still contain complex geometry necessitating complicated connecting paths.
Defining the ease of connectivity is subtle. One natural measure is the parametric complexity of the connecting curves, quantifiable by their degree if polynomial, number of segments if piece-wise. Another possible definition for easy connectivity would be low curvature of the minimum manifold or short geodesic distance between two points it. As we saw in Section 6.2, low curvature implies that linear interpolation stays near the manifold. Other potential definitions include whether the connecting curve has an analytical expression, or how many points are needed to approximate it within a certain error. It would be interesting to examine these properties for symmetry-induced connecting curves.
D. Proofs in Section 5
Lemma 5.1. Consider two points (W1, W2), (W), W3) E (-) that are connected in [.-1 For any g E GT.(h) such that det(g) < 0, g (W1, W2) and (W1, W.) are connected in (0).
Proof. Consider the map and its inverse f-1 defined in (10) in the proof of Proposition 4.1. Let g (W1 W2) and g' = ff (W',, W2). By Corollary A.2, since (W1, W2) and (W1, W3) are not in the same connected component of L- L-)()), 9 and of are not in the same connected component of GL., Equivalently, det(gg') < 0. Consider a 91 E GL such that de) < O. Then de'') > 0, which means that 9.g and g' belong to the same connected component of GL,,- Therefore, according to Corollary A.2, 91 (W1, W2) = fg1g) and (W1, W = f(g') belong to the same connected component of L (0).
1
-1
1) is a point in L-1(0) for some loss function L. Then o
Example.
Suppose
Wi
W2-
1 0
is also a point in (-), However, (W1, W2) and (W1, W3) are not on the same connected component of the minimum, since their determinants have different signs. By Lemma 5.1, any y E GL.(h) with det(g) 0 can bring (W1, W2) and (W,, W?) the same connected component in I. Io- Let g be the permutation
W
W3
1
0
[ 1 Then g (W1, W₂) (D 0 1
1
0
ol)) which is in the same connected component as WW/WW)
matrix
0
Proposition 5.2. Assume that h > 2. For all (W1,.., Wi), (Wi,-.. -.... W/ T,-) these exists list of permutation matri- ces Pt. ..., P1- such that (W.P. P. W2P2 P1 1 PI W)) and (W1,.., W() are connected in L --)
Proof. Let (g1,...,gI-1l) (0), ................................ I E (GL,)"-1 such that f(g .... 91-1)) (W1,.., Wi) and f(g'
.... Ri For i = I,..., 1, if (gggg > 0, set P3 to Otherwise, we set P. to an arbi- trary element in P E Sh \ Ar, which is not empty when 22 2.
(Wi,.., Wi). Let Po
Let (o"...,""'))
PL-WW). By the way we construct Pt's, we have of PT and det(g.g()) > 0. Therefore, Di and gf belong to the
(GLa)"-1 such that flgi,..., 1"11 = (WiP,, P W.PP....PP-WWWWWW
16

---

Understanding Mode Connectivity via Parameter Space Symmetry
same connected component of (GL,)--1 for all i. Since f is homeomorphism between (GL,) and L (0),
(W₂ Pi, 2 P1- P Pi-1Wi) and WW....,y)) are connected in (0).
Proposition 5.3. Consider the loss function of the following
L: Param + R, W = (W,...., Wi) + 1|2 Wio Wi-3..., Wi)
(13)
some where f is a function of Wi-2, Wi-3.... ...y W1, X, and o (cz)) (=) all E R some k >> 0. Assume that || la + O
and (0) # 0. Also assume that 1, 1 2. For any positive number h > 0, there exist W, W E [0) that belong to the same connected component of L (0) and 0 < a < 1, such that L ((1 - a)W + +aW") > b.
Proof. Let W = (Wi,... W2, W1) (0) be arbitrary point on the minimum of L. Let W' = (W7, .... W3, Wi) (Wim mWi-1 Wi-, W)). Then W. W" belong to the same connected component of L (0), connected by curve 7: R- + Param, P4r = ((1- W)W + (Wim (1 ()Wi-1 + 1 21-2,.., Wi).
Since W E L (0), we have Wio
.... W1, Y. The loss on the linear interpolation of W, W' is [[1-a]W; w.] + aW)) - a
L [ - [ + +aW" =||Y - ((1 -
W1, W2, l1
=|YY (1- a1-a+ cmm -) "/ (1-a (1 W/ 1)
=||Y - (1 a1 a V/
a
W1,
=(1 - (1 - 0 + am a + )
(14)
Let a= = 0.5. Then
L ((1 - ((1 + aWW) = 1 2 + m -k x-++t)/
(,
2 m ))² ||8|
-11-2-(411)) (k+1) (1 m )1
(1+ + m)) 2 ||8l1
(15)
Let m = (2h41 (-b TTl +1) -1)* k Recall that O. Then m > 0, (1 + m))* > 1, and
vb
2 (k(1) (1 + m -1 m)* 2 2-(1)) m k
+1>1
(16)
Y
Therefore, the loss chosen values of and is at least b:
TT|P +1 +1)) ([l-aa) + +aW")
(1- (1-(11)) vb
||8 = b.
(17)
Figure 4 visualizes the loss barrier on the linear interpolation between two minima. We construct a network with loss function [|aa
YIJ. with a being leaky ReLU function, X E R Y E R4x4, and (W1, W2, Wa, W4, Ws) E Param R16x8 x R-2216 R16x32 R8x16 R4x8 The network is initialized with random weights, and each clement of X, Y is sampled independently from normal distribution.
We obtain the first minima (Wy, WT, WA, W4, Wg) by SGD, and the second (W W W WT, WT) (W1, W2, W/. mW" 11 -1m by rescaling the last two layers with m E R+ At large m, the two minima are farther apart, and the loss evaluated at the middle point of their linear interpolation grows unboundedly as predicted by Proposition 5.3.
Proposition 5.4. Consider the loss function with the same set of assumptions in Proposition 5.3. Assume additionally that there does not exist a permutation P such that every column of (W W.-...., W))X)) is in the null space of Wi. For any positive number b > 0, there exist (W1..., Wi). (Wi..., W/) E L 1(0 and < a < 1, such that (W1... 2i-2) = (Wi,.., w..,) and minres, L ((1 WW Wi) + <aW,,.., Wi-2, P-1W-1, W,P)) > b.
17

---

12000 Understanding Mode Connectivity via Parameter Space Symmetry
1e6
Barrier height
2
1
0
4000
8000
Rescaling factor m
Figure 4: Loss at the middle of the linear interpolation between two minima in a homogeneous network becomes unbounded when the two minima is far apart.
Proof. Let W = (W,,... 22, Wi) E (0) be an arbitrary point on the minimum of L. Let W = -WW .... W2 W7) (Wim"* -mWW1,, Wi-2. W1).
Since W E -. (0), we have Wio [Wi11 W1, x)] The loss on the linear interpolation of W, W" is
L ((1 - a-) + aWW) =1|Y ((1 W, + awWPPo - a a
W1, x)]x 12. (18)
Let 0.5. Then
L ((1 a aW)) =| - ( x), 1 (19)
When m + 00,
lim a mP hin
m=000
lim n [(m -1 p (W- W)]
m=000
= lim p
W))
(20)
m=000
Therefore,
lim L L((1 a)W + +aW") = lim ||
Wi(l + m m=mmmm p- [Wi-1 (W, X1]
m 100
m00
lim
Y
Wi(I p (Wi-n X)1
m0000
m 1
lim m=000
(21)
Since we assumed that there does not exist a permutation P such that every column of Po Wi-3,.., W,, X)) in the null space of Wi, at least one element in the second term is unbounded for any permutation P. Therefore, L ((1-a)) - a: + aW') unbounded for any P.
Proposition 5.5. Let A E TRxx be an invertible matrix. Let set S = ((W, W2): Wi, W2 E ROX WWW, = A). For any positive number
there exist W W" and 0 < a such that minwes ll ((1 a) +WW") W|l
Proof. Let W be an element of S. Let W/ = Wig W = gi W2, W" = Wigz and W = 92 92WW, where 91, E Ruxn are invertible matrices. Note that W' = (W7, W3) and W = (W"" W are both in S. Then.
min ((1 - W' + aW") - W1l3
WeS
min [[(1 -
[[(1-x)W1s) W1 + +aW19
Will + [1(1 -a) We + og Wo
Wes
min
[Wi((1-a)) +agz g 1) + [| a)g1 ag2 g)12.
(22)
gEGLin)
18

---

Understanding Mode Connectivity via Parameter Space Symmetry
Let 91 3I and 02 /- for some B > Let 2/2 Then, in the limit of large we have
lim min [ a)W + +aW")
W11
8000 WES
3 13+3- 3-1 1 - 9 g-1) ||
88+3- 3-1
Jim 8-000 gEGL(m)
min
W1
+ W2
1 -
(23)
2
2
As B oc, 9 andg-- cannot approach [+.-1 simultaneously. Therefore, (23) is not bounded.
2
Proposition 5.6. Consider the loss function with the same set of assumptions in Proposition 5.3. Let W (0) be point the minimumm. Consider the multiplicative group of positive real numbers Rt that acts L (0) by g. (W1, W)) (W1, Wi-2.gWi-1,W; W.gh"), where E RR. Then there exists a positive number b >> 0, such that for all 0 < a < and W E Orbil(W) with [| < efoor all i and some c > O, the loss value for points on the linear interpolation L ((1 - a)W + +a") < b
Proof. Since W E (W), W = (Wim -h -W 2i.,,..,WW, for some m > 0. Additionally, m and m k are bounded since W is bounded. Since W L (0), we have Wio [Wi-1f(Wi-2,..,, W1, X)] = Y. The loss on the linear interpolation of W, W is
L a + +aW') =||Y ((1 a>Wi + aW/)
ta/ W W1, l1
=| (1 + om Who a1-a+ (w- 1,, W), 112
=||Y - (1 a + am (1 a +mm)* | Wi, X)]
=(1 (1 + am (1 |111))
(24)
As m, m m- and a are all bounded, the loss value for points the linear interpolation L L a)W + +aW" is also bounded.
The connectedness results derived from symmetry raise several interesting questions about mode connectivity. For example, it would be interesting to understand when and why there is no significant change in loss on the linear interpolation between two minima. One possible explanation is that there always exists a caametry-induceed path y that stays close to the linear interpolation. Another potential factor the high dimensionality of the minimum, which increases the likelihood that significant portion of the linear interpolation remains within the low-loss region. Additionally, empirical observations suggest that both train and test accuracy remain nearly constant along paths connecting different SGD solutions (Garipov et al., 2018). If these paths are induced by group action, this would imply that the group action's 's dependence data is weak. Investigating the extent to which data influences these symmetries could provide deeper insights into the structure of the loss landscape and the generalization properties of neural networks.
E. Proofs in Section 6
-9.UU, (U, Then Proposition 6.1. Let (U,V) Param, and V") = g.
[Uo(VXX) U'o(('X)) < [[o[(X)]]
(25)
Proof. We note that 7 1 is projection:
(I x
=I htwp://wwonn-gggooooooo--oo-oooo =1 x(gV
Therefore,
= [ ((oovx(((-o()xx))ooovxx) [[]][
(26)
19

---

Understanding Mode Connectivity via Parameter Space Symmetry
Theorem 6.2. Let L (c) Param, with E R, be level set of the loss function L Param - R. Lety [0, + (c)
be smooth curve in I--1 (c) connecting two points wi (()) and w2 (1). Suppose the curvature x(t) of satisfies k(t) S Kmax for all i E [0.
Let S be the straight line segment connecting wi and w2. Then, for any point w on S, the distance to L (c) is bounded by
(1-V1-
Kimax 122
wt 12
dist(w, (c)) S dmax
Kimax
2
Furthermore, assuming Lipschitz continuous with Lipschitz constant Ct. the loss at any point up on satisfies
[L(w) -cl S Cidmaxx
Proof. We will find an upper bound for the maximum distance between smooth curve and the chord connecting two points on the curve, assuming the curvature of the curve is bounded by 6max-
The curvature x point on a curve defined as K J. where R the radius of the osculating circle at that point. Let be the maximum perpendicular distance from the midpoint of chord to the curve. For circular are, Pythagorean theorem gives
[12
w1
2 + (B - s)?
R²
2
Solving for s:
||22
wi willa
R
1
2R
Substitute
1 into the above, we have x.
(kellez
un w0il12 2
V1
R
2
Since the curvature of 7 is everywhere less than equal Fimax. the cannot bend sharply than the osculating circle with curvature Kmax- Therefore, the maximum deviation dmax between and its chord cannot exceed that of the osculating circle:
dist(w, (c)( & dmax
def
1- 1--1-
Kmax
W22
wi ||
Kimax
2
Assuming is Lipschitz continuous with Lipschitz constant Cr, for any w on S, we have
L(w) - c| = (L(w) - L(r(t)) s Czl w - ((t)ll s Cydmax
20


Reviews:
[{"review_id": "btcWRywZyT", "reviewer": "Reviewer_jnZq", "summary": "This paper look at the topology of the loss level sets in order to understand their connectedness, i.e. mode connectivity. They are specifically investigating the topology of symmetry groups of the weights under the loss. They deduce the number of connected components of full-rank multi-layer linear networks (both with and without skip connections), and prove mode connectivity up to permutation for full-rank linear regressions, as well as derive some instructive examples.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "It is a very sensible approach to use topology to understand the linear connectedness of minima. In terms of linear networks, the results are reasonable and informative. I appreciate the two layer examples and the examples of failure cases going from linear networks to more realistic examples. Perhaps also a more high-level discussion about the limitations of the work would be helpful--what are the main difficulties with non-linearities and how do you expect it to affect the results? Looking for some intuition.", "methods_and_evaluation_criteria": "Yes, it does. I think the paper is quite dense with a fair amount of math. I would appreciate more examples and plots. Especially for an 8 page conference paper, I think that would be helpful.", "theoretical_claims": "I have not gone through the formal verification of each proof, so I cannot assess their correctness in detail. However, the paper’s theoretical approach appears internally consistent and is built on standard techniques from topology and group theory.", "experimental_designs_or_analyses": "The empirical results appear reasonable and align with the paper’s theoretical statements. I did not inspect the implementation details, so I cannot comment on reproducibility or correctness of the code.", "relation_to_broader_scientific_literature": "It looks OK to me, but I am not very familiar with related work. I would appreciate some more references to applied topology within machine learning. Now the related work only focuses on mode connectivity, but topology has been applied to ML problems and to weights of neural networks for some time, which I think deserves brief mentioning, e.g. \"Exposition and Interpretation of the Topology of Neural Networks (2019)\".", "essential_references_not_discussed": "I am not aware of any additional critical references beyond those mentioned, though, as stated, a broader selection of topological approaches in ML might enrich the discussion.", "other_strengths_and_weaknesses": "I think this is a solid paper. Topology is a brilliant and obvious tool to use study connectedness of spaces. I would appreciate some more high-level intuition of how this might generalize to more realistic NNs, as well as more examples and plots in general. However, I have not checked the proofs nor am I confident in addressing the novelty as I am not well-versed in this area.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please see comments above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fIMOH0W8Bb", "reviewer": "Reviewer_7vQu", "summary": "The paper studies the topology of the minima in neural networks through their symmetries. Results of the paper consists of different simplified models of neural networks, for example using single units and single data point, or assuming linear networks. The paper the studies the effect of different types of symmetries – e.g., GL(n) for linear networks, or rescaling symmetry for homogeneous activations – on various properties of the loss landscape – e.g., the number of connected components, linear mode connectivity – in these simplified settings.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. Below, we address concerns about the scope and generalizability of our results, and clarify how our approach can be extended.\n\n**Scope and generalizability of results**\n\n> Sometimes the scope of the findings is inaccurate when a high-level picture is given…\n\nWe will clarify the intended scope to reduce ambiguity. For higher-dimensional invertible weights in the skip-connection example, the number of components reduces further to 2. We are extending the analysis by relaxing the invertibility condition and will include full proofs in the final version.\n\n> The investigation relies on the symmetries of simplified models of neural networks… This limits the scope of finding to toy settings where we know for a fact the conclusions break down when we move to a simple MLP.\n\nWhile our analysis focuses on simplified models, it does not fundamentally break down for more complex architectures. Many modern architectures retain large spaces of continuous symmetry—for example, our results in Section 5.3 apply broadly to layers involving matrix multiplication, such as transformer blocks. When a homeomorphism exists between the symmetry group and the set of minima, topological properties such as connectedness can be inferred directly. In non-invertible settings, such as with skip connections, connectivity can still be analyzed through how orbits intersect, albeit with more care.\n\n> Results rely on strong assumptions, so much so that unfortunately conclusions tell us nothing about realistic neural networks. …one can be sure that conclusions do not transfer to more complicated architectures, since these symmetries do not exist.\n\nWe respectfully disagree that our findings are irrelevant to realistic neural networks. We do not claim direct applicability to all architectures; rather, we aim to isolate the role of symmetry in shaping the loss landscape. Our simplified models allow for precise, mathematically grounded insights into how symmetries–especially continuous ones–can yield connected minima and help explain phenomena like linear mode connectivity.\n\nAs explained above, our methods provide a framework that can be adapted to a wide range of architectures. Even when we do not know the full set of symmetry, a subset of symmetry or even approximate symmetry can still give useful information on the structures of minima and inform new applications, as demonstrated in section 6.\n\n> The conclusions made about the role of symmetry in this paper completely ignores (and eliminates) the effect of overparameterization symmetry…\n\nOverparameterization increases the dimension of the minima set, reflecting redundancy or flatness, but does not induce symmetry groups (e.g., permutations or rescalings) in the group-theoretic sense like the architectural symmetries. Thus, our results neither overlook nor contradict existing theory on overparameterization. Exploring interactions between high-dimensional minima and architectural symmetries remains an interesting direction for future work, and our framework may help analyze such cases.\n\n> Could you please explain if you expect any of the findings to “approximately” translate to a realistic/practical setting of an MLP, or even a linear neural network in the overparameterized regime?\n\nOur main multilayer setting (Equation (1)) already allows for overparameterization. While not all MLPs will exhibit the same topology as our simplified models, we believe our approach is generalizable. For higher-dimensional weights, connectivity analysis involves characterizing intersections among multiple orbits defined by distinct rank combinations. Extending our analysis to include nonlinearity could involve approximate symmetries, as in Section 6, or by continuously deforming the minima and studying its behavior in the limit as the network approaches a linearity.\n\n**Writing and presentation**\n\n> The paper reads like a collection of loosely related results; it includes different toy models used to study different phenomena.\n\nThe paper's central theme is that parameter space symmetries, especially continuous ones, can help explain why and how minima are connected. Each model is chosen to isolate a specific aspect of this symmetry-connectivity relationship. Collectively, they build a coherent picture of how architectural symmetries shape the loss landscape and empirical behaviors observed in deep learning.\n\n> I got lost at the beginning of section 6... It would be nice to clarify the material of that section…\n\nWe will revise this section to clarify and better motivate the use of one-parameter subgroups. The transformation in Equation 6 is not arbitrary–it is specifically constructed to follow a one-parameter subgroup and defines a smooth curve within the minima. The section analyzes the curvature of such symmetry-induced curves to understand when linear mode connectivity approximately holds. Proposition 6.1 formalizes this by relating interpolation loss to curvature.", "claims_and_evidence": "Theoretical claims are supported by proofs in the appendix and enough explanation in the main body to give an idea as to why they are correct. Sometimes the scope of the findings is inaccurate when a high-level picture is given; for example, the paper says that it shows the number of connected components for full-rank linear regression is reduced through skip-connection, but this result is limited to 3-layer networks with single neuron in every layer.", "methods_and_evaluation_criteria": "The investigation relies on the symmetries of simplified models of neural networks; for example, the fact that when we remove all activation functions, neural networks have GL(n) symmetry, and the fact this group has two connected components and it subsumes the permutation symmetry. This limits the scope of finding to toy settings where we know for a fact the conclusions break down when we move to a simple MLP. Nevertheless, the paper has some interesting insights, for example, the fact that the topology of the symmetry group affects the topology of the loss level-sets.", "theoretical_claims": "No, I did not check the proof.", "experimental_designs_or_analyses": "Experiments are justifiably quite limited.", "relation_to_broader_scientific_literature": "The section on related works has a discussion of prior works on mode connectivity and symmetries of the loss landscape. I found the coverage quite adequate.", "essential_references_not_discussed": "Not to my knowledge.", "other_strengths_and_weaknesses": "Strengths\n\n– Nice coverage of related literature\n\n– Clear presentation of necessary background on topology and groups\n\n– Clear statement of results in each section (with the exception of section 6)\n\nWeaknesses\n\n– Results rely on strong assumptions, so much so that unfortunately conclusions tell us nothing about realistic neural networks. Note that as opposed to many toy models, here, when dealing with symmetries, one can be sure that conclusions do not transfer to more complicated architectures, since these symmetries do not exist.\n\n– The paper reads like a collection of loosely related results; it includes different toy models used to study different phenomena.\n\n– The conclusions made about the role of symmetry in this paper completely ignores (and eliminates) the effect of overparameterization symmetry, which has a much larger degree of freedom than symmetries of the architecture and potentially a bigger role to play in the geometry of the landscape.", "other_comments_or_suggestions": "I got lost at the beginning of section 6, after reading equation 6 and proposition 6.1. It would be nice to clarify the material of that section, in particular, why such an arbitrary transformation is used, and what it has to do with a one-parameter subgroup.", "questions_for_authors": "Could you please explain if you expect any of the findings to “approximately” translate to a realistic/practical setting of an MLP, or even a linear neural network in the overparameterized regime?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E9WII4TNKZ", "reviewer": "Reviewer_MTSB", "summary": "This work provides an interesting perspective on mode connectivity by linking topology of symmetry groups to the topology of minima. The key technique used in the paper is based on deriving the number of connected components of minima in linear networks (showing $2^{l-1}$ components for a network with $l$ layers), and in addition they show that skip connections reduce this number (although this is done in a very simplified setting,  e.g., from 4 to 3 in a scalar ResNet). Based on this the authors go on to prove mode connectivity up to permutation for linear two-layer networks with invertible weights, and then go on to show how this fails when considering the multi-layer case. Further, some interesting analysis is carried to find curves on the minimum", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback and insightful questions. We are encouraged by the recognition of our paper’s novelty, rigor, and intuition. Below we expand on the practical takeaways and respond to the questions.\n\n**Practical takeaways**\n\nOur work shows that parameter space symmetries, especially continuous ones, explain why and how minima are connected. For practitioners, these results motivate concrete strategies – and cautions – for tasks that navigate the loss landscape, including model merging, ensembling, and fine-tuning:\n\n- One can build low-loss curves explicitly using known parameter symmetries. This gives a principled and efficient way to obtain new minima from old ones, potentially useful in (1) generating model ensembles with low cost; (2) improving model alignment by allowing a much larger group of transformations than permutation; and (3) mitigating catastrophic forgetting in fine-tuning by constraining updates to remain on the symmetry-induced manifold of the pretraining minimum.\n\n-  The connectedness of minima supports the practice of model merging and ensembling, even when models are trained separately. In addition to permutation, many other symmetry transformations can connect solutions that would otherwise appear very different. \n\n- Linear interpolation between minima is not guaranteed to lead to better models, despite its widespread use. This highlights the need to evaluate whether the minima found by specific learning algorithms are approximately connected before averaging models directly.\n\nFinally, for theorists working in this area, shifting focus from viewing loss landscapes as chaotic to structured, symmetry-rich spaces invites new mathematical tools to explain empirical observations in deep learning. This has led to new intuitions ranging from why residual connections lead to more connected minima, why linear model connectivity can fail, to why model averaging often works when the models are not too far from each other. We also hope that the broader approach – inferring properties of unknown solution sets from known symmetry groups – could inspire new work beyond this field.\n\n**Response to questions**\n\n> Is it clear that having a lower number of connected components makes for easier connectivity? \n\nWe agree with the reviewer that connectedness alone does not imply easy connectivity in the sense of short or simple paths between solutions. Being in the same connected components is a necessary condition for connectivity, but a single component may still contain complex geometry necessitating complicated connecting paths.\n\nDefining the ease of connectivity is subtle, and we agree that a discussion would be valuable. One natural measure is the parametric complexity of the connecting curves, quantifiable by their degree if polynomial, or number of segments if piece-wise. Another possible definition for easy connectivity would be low curvature of the minimum manifold or short geodesic distance between two points on it. As we discussed in Section 6.2, low curvature implies that linear interpolation stays near the manifold. Other potential definitions include whether the connecting curve has an analytical expression, or how many points are needed to approximate it within a certain error. It would indeed be interesting to examine these properties for symmetry-induced connecting curves.\n\n> How do you reconcile the failure case of multi-layers with the empirically observed largely well-connected paths for deep networks?\n\nThe empirical observation of mode connectivity and linear mode connectivity is likely due to the fact that certain family of optimizers, especially stochastic gradient descent (SGD), tend to explore only a subset of the minimum, a phenomenon often referred to as implicit bias. Regularization techniques and weight decay may further encourage SGD to favor certain regions of the minimum. As a result, the subset of minima that is likely to be reached by SGD can have very different structures and perhaps more connected than the full set of minima.\n\n> I am curious what are the nature of theoretical bottlenecks that impede a general analysis for ResNets, besides the general case.\n\nExtending our proof to a general analysis is challenging because the number of orbits grows rapidly with the dimension of the weight matrices. In the 1D case, the minimum consists of only two orbits, so the proof only requires analyzing the connected components of each and checking how they intersect. For higher-dimensional weights, especially when not full-rank, each distinct rank combination defines multiple orbits, and characterizing the intersections among these many orbits, while possible, becomes combinatorially complex. We therefore chose to include only the 1D ResNet example to demonstrate the proof technique. It is possible that future work with more refined tools could extend this analysis to higher-dimensional settings more efficiently.", "claims_and_evidence": "Yes, more or less. But see the later discussion in questions or weaknesses, where there are some questions on how these results fit in to the empirical behaviour of LMC.", "methods_and_evaluation_criteria": "See the strengths and weaknesses", "theoretical_claims": "no", "experimental_designs_or_analyses": "See the strengths and weaknesses", "relation_to_broader_scientific_literature": "The paper builds on and extends prior work on mode connectivity and loss landscapes. The key contribution—linking symmetry group topology to minima structure—is novel and brings in a complementary and rigorous perspective.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "Strengths:\n- The theoretical connections made are quite interesting and might lead to more such works from the topological perspective.\n- The accompanying setups and figures provide useful intuition to think about the problem.\n- The paper is quite well written.\n\nWeaknesses: \n- Despite the above, the practical takeaways practitioners or those working in the area can have remain vague at best.", "other_comments_or_suggestions": "n/a", "questions_for_authors": "- Is it clear that having a lower number of connected components makes for easier connectivity? I am not so sure, as even in a single connected space, it might require a complicated path. Maybe it would be nice to have some sort of discussion in the paper, as what would be ideal atleast from a topological perspective to connect the network solutions. \n\n- How do you reconcile the failure case of multi-layers with the empirically observed largely well-connected paths for deep networks?\n\n- I am curious what are the nature of theoretical bottlenecks that impede a general analysis for ResNets, besides the general case.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mODXIGkejm", "reviewer": "Reviewer_GSK8", "summary": "This paper provides a group theory framework to study the connectivity and the number of components of the zero-loss set in a (non-convex) loss landscape. A precise characterization of the number of components and connectivity of the zero-loss set is given for deep linear neural networks. Some results on the existence of a high-loss barrier are given for non-linear neural networks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their encouraging feedback and insights on our work’s relation to broader literature. We will incorporate their suggestions into the final version of the paper. \n\n> Some well-known simple things are presented as Propositions (for ex. Propositions 3.4 and 3.5). These are not particular contributions of this paper hence should not be presented as Propositions.\n\nSection 3 is intended as a background section, and it is not our intention to present these propositions as novel results. We chose to include them in proposition form primarily for completeness and ease of referencing them in proofs. We agree they should not be positioned as contributions of this paper, and we will revise the paper to clarify this.\n\n> ResNet 1D section should not be called that as that architecture has no non-linearity.\n\nThank you for pointing this out. We will change the terminology to skip-connection to avoid ambiguity.\n\n> Comment: Proposition 5.2 uses permutations to connect components for a deep ---linear--- network. However, the relevant symmetry group here is $GL_n$ as those matrices can be multiplied by any invertible matrix without changing the loss. In other words, symmetries of deep linear networks have more ---volume--- compared to the deep non-linear networks.\n\n> Please add a comment along these lines so that the readers are aware of this distinction between linear and non-linear networks.\n\nWe appreciate this observation and will include a comment along these lines. While permutations are sufficient to connect components in deep linear networks, the full symmetry group in this case is indeed $GL_n$. Also, it is indeed true that deep linear networks have larger symmetry groups than deep non-linear networks.", "claims_and_evidence": "It is fairly clear. Some well-known simple things are presented as Propositions (for ex. Propositions 3.4 and 3.5). These are not particular contributions of this paper hence should not be presented as Propositions.", "methods_and_evaluation_criteria": "NA", "theoretical_claims": "I did not find a mistake in Theorems.", "experimental_designs_or_analyses": "NA.", "relation_to_broader_scientific_literature": "Linear mode connectivity of deep neural networks is an interesting and active area of research with implications for mode connectivity and model merging. A complete and group-theory-based characterization of the zero-loss set of parameters for deep linear networks is a nice contribution to the literature (but I have some reservations about the symmetry-group relevant for deep linear networks).\n\nMoreover, the existence of a big barrier is characterized. This is expected for the distant components which would be rare compared to the close components which would be more common. SGD likely finds more common components which explain the observed phenomenon of linear connectivity.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strength: Characterization of the zero-loss set of deep linear networks is novel and group-theoretic background is the way forward. \n\nWeakness: ResNet 1D section should not be called that as that architecture has no non-linearity.", "other_comments_or_suggestions": "```Param``` should be called $\\Theta$.", "questions_for_authors": "Comment: Proposition 5.2 uses permutations to connect components for a deep ---linear--- network. However, the relevant symmetry group here is $GL_n$ as those matrices can be multiplied by any invertible matrix without changing the loss. In other words, symmetries of deep linear networks have more ---volume--- compared to the deep non-linear networks. \n\nPlease add a comment along these lines so that the readers are aware of this distinction between linear and non-linear networks.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q96O81mUeN", "reviewer": "Reviewer_ZqPy", "summary": "The paper investigates (linear) mode connectivity of neural networks via symmetries in parameter space. Besides quantifying the number of connected components of invertible linear networks with and without skip connections, it sheds light on when modes can be connected linearly by investigating the difference of the linear path between two modes and the orbit between the two points.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their encouraging feedback. We appreciate that they have taken the time to read our proofs. We also appreciate the valuable questions and the many relevant pointers to related work, which we discuss below.\n\n**Relation to broader scientific literature**\n\n> It would be interesting to discuss how the studied symmetries of the general linear group relate to the symmetries discussed by Petzka et al. [1].\n\nPetzka et al. show that permutation and rescaling are the only transformations that leave a 2-layer ReLU network unchanged, excluding degenerate cases. These symmetries are subgroups of the general linear symmetry group studied in our work. They also identify degenerate-case transformations not present in linear networks, when neurons share identical zero hyperplanes.\n\n> How do the loss-preserving orbits discussed in this paper relate to the minima manifold in [2]?\n\nBoth our paper and Simsek et al. [2] analyze symmetry-induced structures in the loss landscape. Our orbits arise from continuous symmetry groups (e.g., GL(n), rescalings), while their minima manifold arises by embedding smaller networks into overparameterized ones via permutations and neuron duplications. These manifolds can be viewed as unions of affine subspaces, and our orbits form structured subsets within it. The two views are complementary and compatible, especially in linear networks with full-rank weights.\n\n> This paper discusses linear reparameterizations. How does it relate to works on non-linear reparameterizations using Riemannian geometry [3]?\n\nKristiadi et al. [3] study reparameterization as a map from the parameter space to a new space, while we focus on symmetry that maps from the parameter space to itself. Both maps can be nonlinear. Studying symmetry-induced orbits from a Riemannian perspective could be a promising direction for future work.\n\nWe will include a brief discussion of these connections in the final version of the paper.\n\n\n**Limitation of the analysis**\n\n> most of the analysis is for linear neural networks and thus the applicability to modern deep learning is limited. While a detailed empirical analysis of this difference might be out of scope, a discussion of these limitations seems appropriate.\n\nWe appreciate the suggestion and agree that a discussion would be valuable. Our results are derived under architectural assumptions that enable precise, mathematically grounded insights into how continuous symmetries shape the loss landscape. \n\nWhile the findings may not directly transfer to all architectures, our methods are adaptable. For example, Section 5.3 applies to any network with layers involving multiplication of two weight matrices–a pattern that appears, for example, in transformer blocks. In higher-dimensional, non full-rank settings, connectivity depends on how orbits defined by different rank combinations intersect. Extensions to nonlinear networks are possible by identifying approximate symmetries (as in Section 6), or by continuously deforming the minima and studying its behavior in the limit as the network approaches a linear regime.Even partial or approximate symmetry can provide useful structural information about the minima and support new applications.\n\n**Other comments and questions**\n\n> Would, for example the proposed relative flatness [5] be invariant under the reparameterizations discussed in this paper?\n\nYes, in many cases. The relative flatness measure proposed in Petzka et al. [5] is designed to be invariant under layer-wise and neuron-wise linear reparameterizations, such as rescaling and certain changes of basis. These are precisely the types of continuous architectural symmetries we analyze in our paper. Therefore, for the linear and homogeneous network settings considered in our work, the relative flatness measure would indeed remain invariant under the reparameterizations we describe. \n\n> Prop. 5.4: in the equivalence of weights $W_i$ and $W_i'$ at the end of the paragraph, the permutation $P$ seems to be missing.\n\nThis is indeed a typo. We will add the permutations. Thank you for the detailed read.\n\n> Q1: Cor. 4.2 shows that the number of connected components increases with depth, whereas width plays no role. This is interesting. Can you comment on potential implications for neural network training? It seems related to the fact that deeper networks can be trained easier than wide and shallow ones.\n\nWe do not have concrete results on whether more connected components correlates with easier training, but below we share some speculations. A larger number of connected components could mean the minima is distributed more widely in the parameter space, which makes them easier to find during training. However, an exact analysis would require quantifying the area (or volume) and the exact distribution of the minima, as well as the geometry in the surrounding region. Nevertheless, connecting connectedness and optimization seems to be an exciting line of future work.", "claims_and_evidence": "The paper investigates the connection between symmetries in parameter space and mode connectivity theoretically. All theoretical claims are rigorously proven. Some claims (e.g., connected components w/o skip connections and loss preservation) are supported empirically.", "methods_and_evaluation_criteria": "The paper's contributions are theoretical, yet a few claims are supported by empirical evidence. Where this is the case, the methods and criteria are sound.", "theoretical_claims": "I have checked the theoretical claims up to my abilities. The proofs presented in the appendix are clear and sound.", "experimental_designs_or_analyses": "As stated above, the main contributions of the paper are theoretical. When empirical evidence was presented, the experimental design was sound.", "relation_to_broader_scientific_literature": "- It would be interesting to discuss how the studied symmetries of the general linear group relate to the symmetries discussed by Petzka et al. [1].\n- How do the loss-preserving orbits discussed in this paper relate to the minima manifold in [2]?\n- This paper discusses linear reparameterizations. How does it relate to works on non-linear reparameterizations using Riemannian geometry [3]?\n\n[1] Petzka, Henning, Martin Trimmel, and Cristian Sminchisescu. \"Notes on the symmetries of 2-layer relu-networks.\" Proceedings of the northern lights deep learning workshop. Vol. 1. 2020.\n[2] Simsek, Berfin, et al. \"Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances.\" International Conference on Machine Learning. PMLR, 2021.\n[3] Kristiadi, Agustinus, Felix Dangel, and Philipp Hennig. \"The geometry of neural nets' parameter spaces under reparametrization.\" Advances in Neural Information Processing Systems 36 (2023): 17669-17688.", "essential_references_not_discussed": "To the best of my knowledge, the paper references all essential works.", "other_strengths_and_weaknesses": "The paper is very well-written, the claims are clearly presented, and the proofs are sound. One thing that could be improved is a discussion on the limitations of the analysis: most of the analysis is for linear neural networks and thus the applicability to modern deep learning is limited. While a detailed empirical analysis of this difference might be out of scope, a discussion of these limitations seems appropriate.", "other_comments_or_suggestions": "- It is a bit of a long shot, but in learning theory for neural networks, one problem is that some generalization measures change through reparameterizations. Therefore, reparameterization-invariant measures have been proposed [4,5] and some can even be theoretically connected to generalization. Would, for example the proposed relative flatness [5] be invariant under the reparameterizations discussed in this paper? \n- Prop. 5.4: in the equivalence of weights $W_i$ and $W'_i$ at the end of the paragraph, the permutation $P$ seems to be missing.\n\n[4] Tsuzuku, Yusuke, Issei Sato, and Masashi Sugiyama. \"Normalized flat minima: Exploring scale invariant definition of flat minima for neural networks using pac-bayesian analysis.\" International Conference on Machine Learning. PMLR, 2020.\n[5] Petzka, Henning, et al. \"Relative flatness and generalization.\" Advances in neural information processing systems 34 (2021): 18420-18432.\n\n######### After Rebuttal ############\n\nI maintain my positive assessment and recommend acceptance.", "questions_for_authors": "Q1: Cor. 4.2 shows that the number of connected components increases with depth, whereas width plays no role. This is interesting. Can you comment on potential implications for neural network training? It seems related to the fact that deeper networks can be trained easier than wide and shallow ones.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a careful, thorough, and rigorous study of the connectivity of minima of (certain simplified) neural networks under the symmetries in the parameter space. It presents both theoretical results and some empirical evidence. The reviewers recognize the novelty and the importance of the insights gained by the work. A number of questions were brought up in the reviews, which the authors carefully addressed in the rebuttals. The primary reservation (most clearly articulated by the reviewer recommending a "weak reject", but mentioned by others) is the limited setting of the analyzed models, which have been simplified to allow for rigorous theoretical analysis. But as one of the reviewers points out, it's precisely this simplified setting that allows for proving non-trivial results, and the paper stands as a useful theoretical step in the broader program of understanding the geometry of loss landscapes. The reviewers also appreciate the honesty of scope and clarity of presentation.
Decision: Accept (poster)