A Tale of Two Symmetries: Exploring the Loss Landscape of Equivariant Models
Authors: YuQing Xie
Tess Smidt
Venue: NeurIPS 2025 poster ()
Keywords: equivariance
loss landscape
optimization
group theory
parameter symmetry
symmetry
Primary area: theory
TL;DR: We prove that the interaction of parameter symmetry and equivariance constraints can create critical points and minima in the loss landscape.

Abstract: Equivariant neural networks have proven to be effective for tasks with known underlying symmetries. However, optimizing equivariant networks can be tricky and best training practices are less established than for standard networks. In particular, recent works have found small training benefits from relaxing equivariance constraints. This raises the question: do equivariance constraints introduce fundamental obstacles to optimization? Or do they simply require different hyperparameter tuning? In this work, we investigate this question through a theoretical analysis of the loss landscape geometry. We focus on networks built using permutation representations, which we can view as a subset of unconstrained MLPs. Importantly, we show that the parameter symmetries of the unconstrained model has nontrivial effects on the loss landscape of the equivariant subspace and under certain conditions can provably prevent learning of the global minima. Further, we empirically demonstrate in such cases, relaxing to an unconstrained MLP can sometimes solve the issue. Interestingly, the weights eventually found via relaxation corresponds to a different choice of group representation in the hidden layer. From this, we draw 3 key takeaways. (1) By viewing the unconstrained version of an architecture, we can uncover hidden parameter symmetries which were broken by choice of constraint enforcement (2) Hidden symmetries give important insights on loss landscapes and can induce critical points and even minima (3) Hidden symmetry induced minima can sometimes be escaped by constraint relaxation and we observe the network jumps to a different choice of constraint enforcement. Effective equivariance relaxation may require rethinking the fixed choice of group representation in the hidden layers.

A Tale of Two Symmetries: Exploring the Loss Landscape of Equivariant Models
YuQing Xie Massachusetts Institute of Technology Cambridge, MA 02139 xyuqingemit. edu
Tess Smidt
Massachusetts Institute of Technology Cambridge, MA 02139 temiid@@it.eeee
Abstract
Equivariant neural networks have proven to be effective for tasks with known underlying symmetries. However, optimizing equivariant networks can be tricky and best training practices are less established than for standard networks. In particular, recent works have found small training benefits from relaxing equivari- ance constraints. This raises the question: do equivariance constraints introduce fundamental obstacles optimization? Or do they simply require different hyper- parameter tuning? In this work, we investigate this question through theoretical analysis of the loss landscape geometry. We focus on networks built using per- mutation representations, which we can view as subset of unconstrained MIPs. Importantly, we show that the parameter symmetries of the unconstrained model has nontrivial effects on the loss landscape of the equivariant subspace and under certain conditions can provably prevent learning of the global minima. Further, we empirically demonstrate in such cases, relaxing to unconstrained MLP can sometimes solve the issue. Interestingly, the weights eventually found via relax- ation corresponds to different choice of group representation in the hidden layer. From this, we draw key takeaways. (1) By viewing the unconstrained version uncover hidden parameter symmetries which were broken by choice of constraint enforcement (2) Hidden symmetries give important insights loss landscapes and can induce critical points and even minima (3) Hidden symmetry induced minima can sometimes be escaped by constraint relaxation and observe the network jumps different choice of constraint enforcement. Effective equivariance relaxation may require rethinking the fixed choice of group representation in the hidden layers.
architecture,
Introduction
the best of models, it was the worst of models, was the age of augmentation, was the age of equivariance...
Many domains possess latent symmetries, inspiring the development of equivariant networks which
by design already account for these symmetries [12 [3] 47 53 [5]]. Such networks have proven to
be extremely effective in a wide variety of tasks including molecular force fields (6] [ (7], catalyst discovery [35], charge density prediction
(23), generative models [27], protein structure prediction 29 33 and particle physics 9 10]. One of the major benefits of equivariance is improved sample
complexity, and better generalizability. both of which have concrete theoretical support g [9] 36
However, it also has been observed that equivariant networks can be tricky optimize and best training practices for equivariant networks are less established [30] 35 51) addition, there have been number of works exploring the relaxation of equivariance constraints [ 49 50 [2]. The motivation for these works is model misspecification. Real world data may not have the perfect
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

symmetry assumed in equivariant models, so it could even be harmful to have perfect equivariance. Indeed, relaxed models do appear have small performance benefit over perfectly equivariant models 49 52
and can have better theoretical performance on such data 43 50
Combining these observations, natural question arises, relaxing equivariance improve training even when we expect a perfectly equivariant final solution? This question is addressed in [41]. In this work, an additional unconstrained linear contribution was added the equivariant layers and regularization terms were added to the loss to still encourage equivariance. At the end of training, this additional contribution is removed, projecting the model back an equivariant subspace. Empirically, they demonstrate this procedure improves training performance on variety of equivariant models. While compelling, the improvements are often small and there is little theoretical insight on how specifically the relaxation helps.
In work, seek address themselves create fundamental obstacles to optimization? In this paper, we show that our choice of constraint enforcement can hide parameter symmetries of the larger unconstrained model class and that these hidden symmetries can induce critical points and even spurious minima.
more foundational question: Do equivariance constraints
We structure our paper as follows. In Section 2 we introduce the key concepts related to equivariance and parameter symmetry needed for our work. In Section we apply these concepts to analyze the loss landscape. In particular, function space can create critical points and even spurious minima for the constrained model. We then apply these findings equívariant networks and characterize their hidden fixed point subspaces. In Section 4 we empirically explore loss landscapes theoretical findings of spurious local minima and empirically demonstrate the benefits of relaxation.
prove that parameter symmetries of models in unconstrained
teacher-student setup. We validate our
In particular, the relaxed model chooses a different group representation for its hidden layer.
Unconstrained
Linconstrained models
models +
ÔH
models
modes
he
(a)
(b)
Figure 1: (a) Standard viewpoint of why constraints may help. we know the ground truth function satisfies some type of constraint, then enforcing such constraints may reduce the search space allowing for better training. (b) In reality, unconstrained networks contain many parameter symmetries. However, enforcing constraints (such as equivariance) require choices that breaks some of these symmetries. These broken symmetries are "hidden" to the constrained model but still influence gradient values and in some create spurious minima and hinder learning. Further, empirically observe that when completely relaxing equivariance constraints, network can sometimes "jump" between subspaces corresponding different ways of enforcing the same type of constraint.
As a result, we find three key takeaways from our work:
By viewing given architecture as constrained version of larger function class, we can uncover previously hidden parameter symmetries that were broken by our choice of constraint enforcement.
Hidden symmetries have important consequences on loss landscape and can induce critical points and even minima.
Empirically, we find a spurious minimum in an equivariantly constrained subspace can escape to a better minimum in symmetrically related subspace through constraint relaxation. To achieve more effective relaxation of equivariant models, one may have to rethink our fixed choice of group representation in the hidden layers

---

2
Background
Here, we introduce the key concepts necessary for our work. A more in depth overview can be found in Appendix C
including an overview of necessary group theory.
2.1 Equivariant networks
We first formally define equivariance.
Definition 2.1 (Equivariance). Let G be a group with some group action on spaces X, Y. function f: Y G-equivariant if for all g E G, X, y E Y we have f(gz) = ff
Essentially transforming the input is the same transforming the output of an equivariant function. Importantly, the composition of G-equivariant functions is still G-equivariant. Hence in practice one often implements equivariant neural networks by designing equivariant layers and composing them.
2.1.1 Group representations and irreps
One usually works with vector spaces and linear group actions. Such an action referred to as group representation.
Definition 2.2 (Group representationn. Let G a group and a vector space. A group representation a homomorphism p G + GL(VV.
In many cases, we can explicitly decompose representations into direct sum of representations on smaller subspaces. Representations which cannot be decomposed further are irreps.
Definition 2.3 (Irreducible representation). Let G' be a group and V a vector space. A group representation p G > GL) is irreducible if there is no subspace W c V other than 0 such that gW = W for all G.
The irreps are well understood for many groups. particular, Schur's lemma gives an easy way to parameterize maximally expressive equivariant linear layers once we have an irrep decomposition.
2.1.2
Nonlinearities and permutation representations
There have been a number of studies analyzing the construction of equívariant networks 212 14] 26 53). Our work focuses on networks which use permutation representations. These include group convolutions (12). Deep Sets (54). and spherical signals [13. The reason for working with permutation representations is that they are the only type of representation compatible with arbitrary pointwise nonlinearities (39]. Such networks can be viewed as subspace of unconstrained MLPs, which are relatively well studied.
Definition 2.4 (Permutation representation). Let be a group. Any group homomorphism π T:G. Sym(n) is abstract permutation representation. Let p Sym(n) - GL(V,, map corresponding permutation matrices acting on n-dimensional vector space Vn. Then o म G - GL(VV) ) is a permutation representation. We will abbreviate permutation representation p-rep and p
p.
The intuition is that for a p-rep, there is always some choice of basis by such that p(g)b; = b In other words, group actions just permute the basis. The trivial representation is example since there is only one basis and it remains invariant under group action. The regular representation commonly used in group convolutions is another example since
associate each basis vector group we discuss how to generalize p-reps to infinite dimensions.
element 12 In Appendix C.I
Just like how irreps are building blocks of arbitrary representations, are interested in the building blocks of p-reps. Such p-reps are the transitive p-reps.
Definition 2.5 (Transitive permutation representation). Let G' be group and p G GL((V,) be a permutation representation. We say the permutation representation is transitive if there is a basis b 6n, of V, such that the action of G the (bI,..., b,, is transitive.
It turns out that there is a nice way to characterize all possible transitive p-reps. Lemma 2.6. Let G be group. There a canonical bijection between the transitive permutation representations of G and the classes of conjugate subgroups.
3

---

Another commonly used nonlinearity is a tensor product. Given any pair of representations, we can always form new representation by taking the tensor product of the vector spaces. The mapping from the pair of representations into the tensor product space is bilinearity. Tensor products are often used in irrep based equivariant frameworks such as e3nn [26]. It turns out for many groups, there is a correspondence between pointwise nonlinearities and tensor products so our results may be relevant in those architectures as well.
2.2 Parameter symmetries and loss landscapes
Loss landscapes have been studied in numerous works and provides important insight on training
46 Of particular interest for us the impact of parameter symmetries on loss landscape geometry. In M. it was shown that permutation symmetries can induce critical points creating saddles in the loss landscape. This was further explored in [46] where permutation symmetries induce entire affine subspaces of global minima subspaces (which they call an expansion manifold) and symmetry induced critical subspaces. These ideas are complementary
and generalization performance 2 [1 15
16
20
22 25
work.
First, we define what we mean by parameter symmetry. particular, we highlight that we should distinguish these into local and global parameter symmetries.
Definition 2.7 (Parameter symmetries), Let Fe = [Jo X YY θ} be a class of functions parameterized by 0 è where θ is the vector space of our parameters. Any homeomorphism : θ + θ is local parameter symmetry at 0 if fe fre and we denote the group of such transformations as Pg. If p local parameter symmetry for all 0 θ then we say p is global parameter symmetry and we denote the group of such transformations as P.
For example, permuting neurons a MLP gives a global parameter symmetry as this does not change the output of the network. Permuting the output weights of two neurons is local parameter symmetry those two neurons share the same input weights. However, if the input weights are different, then permuting the output weights generically gives different functions this is not a global parameter symmetry. We will focus on the effects of global parameter symmetries this paper.
Next, given any pair of outputs, we choose loss function E: Y x Y - R to measure the discrepancy. Given distribution Dx,y of input-output pairs, we can use the loss function to define a loss landscape as function 7. e -> IR. given by T.(l) Ex.yy-xxyy y [f(fo (z). y)]. Note that by construction, the loss landscape invariant under global parameter symmetries and has the local parameter symmetries Po at any given 6.
Of particular interest for us are spaces which are highly symmetric. Definition 2.8 (Fixed point subspace). Let G be group with linear action some vector space V. Let H < G be a subgroup. Then the fixed point subspace is
Vn = {v E he = e th E H}.
For us, we usually consider θ as our vector space and subgroups of the global parameter symmetries H S P with a linear action. Most commonly considered parameter symmetries act linearly on parameter space.
Fixed point subspaces are useful because of the principle of symmetric criticality which has widespread usage in physics (40].
Proposition 2.9 (Principle of symmetric criticality). Let H be a group which acts on a manifold M through diffeomorphisms. Define
\- -p= Vg E H1..
Let f: : +
be smooth function invariant under H. If E is submanifold, then critical points of fls are critical points of
We see that the loss landscape is invariant under P and in particular, invariant under any subgroup of P. For example, if we consider the subgroup H corresponding to permuting two specific neurons, the corresponding fixed point subspace Oи corresponds to shared input and output weights for those two

---

neurons. This is effectively a network with one fewer neuron. The principle of symmetric criticality then tells us minima of L/e,, is a critical point L, reminiscent of previously known results relating minima of smaller networks to critical points of larger ones [1] 24 46) 'To connect directly back to these prior works, we note that any transformation of the output weights of these two neurons which leaves the sum unchanged is a local symmetry of Ø Hence taking the orbit of Ö gives larger subspace corresponding to the different ways we may embed the smaller network in a larger one.
(a) Permuting neurons (b) Fixed point subspace
(c) Reducible neuron
(d) Smaller network
Figure 2: (a) Subgroup of global parameter symmetry which permutes two neurons. (b) Corresponding fixed point subspace where both input and output weights of these two neurons are the same. (c) There is additional local parameter symmetry the sum of output weights of the neurons is the same, we compute the same function. (d) Smaller network with fewer neuron computing the same function.
the output weights. In particular, long as
It will be useful to also define the "generic" regions of parameter space where we cannot reduce our network into smaller one. If all ingoing weights of two nodes the same, then can merge these two nodes and create single node with the shared ingoing weights and outgoing weights which is sum of the outgoing weights of the original two nodes. Further, any node which has all 0 outgoing weights can be eliminated from the network. We if we cannot perform such reductions, then we say the weights are irreducible 46
Definition 2.10 (Irreducible neurons). We call block of m-neurons in the same layer irreducible if all m ingoing weights are distinct and nonzero, and the outgoing weight vectors are nonzero.
Loss landscape of equivariant networks
Here present the main theoretical results and intuition for why they true. Proofs can be found in Appendix D We first make some general statements that apply to general linearly constrained models, not just equivariant ones.
3.1 Hidden symmetries in constrained models
The key insight is that global parameter symmetries can disappear once we restrict to constrained subspace. However, these symmetries still impact the loss landscape of the constrained model.
If the unconstrained parameter space is θ, let us denote the linearly constrained subspace by Ô. Then of course any p acting on any 0 E θ leaves the resulting function unchanged. However, the problem is that we may have på & é. Hence, such symmetries may be overlooked when analyzing the loss landscape. We call such symmetries "hidden" symmetries.
Definition 3.1 (Hidden symmetries). Let Fe be class of functions parameterized by O. Let P be the group of global parameter symmetries. Let Ô C θ be subspace of parameters satisfying some constraint and the corresponding class of functions Fg. Let F& have global parameter symmetries P. We define a hidden symmetry as any group H where < and & P.
However these hidden symmetries still have consequences the loss landscape of the constrained model. Intuitively, we may try imagining the gradient flow. In particular, assuming nicely chosen parameterization (3I, once we reach fixed point subspace we cannot leave it. For instance, once two neurons share the same weights, they would both receive the same gradient updates and would always share the same weights. But the constrained parameter space can intersect fixed point subspace corresponding hidden symmetry giving what call a hidden fixed point space
5

---

Øz But we can never leave Oµ through gradient descent. In particular, critical points of the loss landscape Llø would be critical points of Lia-
Lemma 3.2 (Hidden symmetry induced critical points), Let Fe C Fe be a class of constrained networks. Let L θ + R be a loss associated to these networks. Let H be hidden symmetry with orthogonal action and let Øh {0 θ h0 = 0 vh H1 and θH θ n
If (0 n () + 1 n ) = é, then critical points of Lle, are also critical points of L
The lemma requires an orthogonality condition (O) nô) + (O h n 6) -. This condition holds in the case of equivariant networks when we reduce a transitive p-rep to smaller transitive p-rep and a relatively mild condition.
can be helpful for understanding scaling While not explored in this work, we suspect Lemma 3.2
behavior of equivariant networks using loss landscape geometry arguments similar to [46..
If the hidden fixed point subspace θи is dimension lower than the constrained subspace, have even stronger statement. In this case, θ in some sense splits θ into two halves. We cannot traverse between the two halves via gradient descent because once reach θ must get stuck. Generically, we expect the global minima to be in one of these halves so if we are initialized in the "bad" half, then would never reach the global minima. This is depicted in Figure Hence, expect there to be an additional minimum on the other side.
To formally state the theorem, we require the following conditions and discuss their satisfiability.
Conditions:
CI ( )) (O n ))
This used to invoke Lemma 3.2
This condition can hold in the case of equivariant networks when neuron permutation symmetries can reduce a transitive p-rep to smaller transitive p-rep.
C2 dim dim(@ dim(@) -
This a strong condition. We expect this condition can sometimes hold in the first layer but will generally fail if the previous layer is wide.
C3 There exists some C such that for any direction ê we have f. This condition will be satisfied if we include regularization. This is mainly used to draw region with a minimum its interior.
(C((F) >
C4 The minimum of L is nondegenerate in Lie This is primarily used to ensure we can fully characterize saddle points and minima via the Hessian. When we have degeneracy (singular Hessian matrix), may result from some additional symmetry which we can exploit to preserve the theorem.
Theorem 3.3 (Hidden symmetry induced minima). Let Fe C Fe be class of constrained networks. Let + R be a loss associated to these networks. Let H be a hidden symmetry with orthogonal action and θH = (0 E O he 0 th E H3 and θ OH 0 ②
Suppose there is minima 01 of Llg such that 02 & ②.. If the conditions above hold, there must exist a distinct minimum 02 + 01 of Lls.
We would like to emphasize that the parameter space being split into two halves by fixed point subspace is not considered a problem. This is because the two halves of parameter space are related by parameter symmetry and represent the same set possible functions. So the extra minima could just be another global minima reached through parameter symmetry. However, for the constrained space, the two halves can arise from a hidden parameter symmetry. In this case, the two halves of the constrained space are not necessarily related by some other global parameter symmetry and the additional minima could truly be a spurious onc. In Section 4 we show this can indeed be the case for equivariant networks.
3.2 Hidden fixed point spaces of equivariant networks
We now characterize hidden fixed point spaces of networks built using p-reps. In particular, we consider the parameter symmetries corresponding to permuting neurons. In fact, we provide slightly

---

= θ 0 n
Figure 3: Constrained subspace split into two halves by intersection with fixed point subspace of the unconstrained model. We call the subspace separating the constrained space a hidden fixed point subspace. Gradient flows from the two halves cannot reach each other implying existence of separate minima.
more general characterization of not just the hidden fixed point spaces but the hidden "reducible" ones based on the concept of "irreducible" neurons from [46].
First, we show that for a given transitive block, any hidden fixed point subspace corresponds to a smaller transitive p-rep. We expect this scenario could negatively impact optimization.
Theorem 3.4. Let a m-neuron point be a transitive permutation representation. Suppose the weights satisfy the corresponding equivariance constraints. If this m-neuron point is reducible, then either we can eliminate these neurons or we can replace the transitive permutation representation with n-neuron transitive permutaionn representation where and n/m.
(a)
(b)
Figure 4: (a) If two neurons within a block of transitive p-rep share input weights, then multiple other pairs of neurons must also share the same input weights. The p-rep can be reduced into smaller one. (b) Suppose two blocks of neurons each individually are irreducible neurons and transitive p-reps. Then if two neurons share input weights, then input weights for all other neurons are shared. The two transitive p-reps can be compressed into one transitive p-rep.
Next, we show that two transitive p-reps which share neurons must collapse to effectively one transitive p-rep. It turns out this same subspace also a fixed point subspace corresponding to permuting the entire two blocks. Hence, we do not expect it to negatively impact optimization.
Theorem 3.5.
Let there be transitive p-reps defined on distinct blocks of m-neurons and mn' -neurnns in layer f, each of which is irreducible. Suppose the m m' neeron point combining these neurons reducible. Then they must both the same permutation representation and we can replace the + m' neuron block with m-neuron point.

---

Experiments
For our experiments, we consider a teacher-student setup so we can guarantee perfect loss achiev- able. We use 2-layer networks with trainable weights only between the input and hidden layer. These are the form
(x: - =o
Σ B5
where a is our nonlinearity, For the experiments here, we use ReLU. Results with other activations be on a laptop
can be found in Appendix All of our experiments
We assume data for each of our input neurons draw independently random from unit normal distribution. We use the teacher networks to label the corresponding output values. For such a setup, turns out is possible to analytically calculate the exact loss landscape for certain activation functions 48 For our loss landscape visualization, we use this exact loss landscape.
4.1 Permutation equivariance
In this section, we consider permutation symmetry S,, and our example uses S3. Appendix we also consider spherical signals used in O(3) and SO(3) equivariant networks.
We consider 2 types of p-reps. First, have the trivial representation To which is 1-dimensional. We then have the usual action of Sn which permutes orthogonal basis vectors, which gives a n--dimensional representation Tn- In our code, we also have support for the regular representation Treg which [S = vil-dimensional and is found group convolution
Between No and Tni it is not hard to check the only possible equivariant linear map is of form 01 where 1 is n x matrix consisting only I's. Between π and Ti, possible equivariant maps are of the form 0, I + 0,(11' L 7) where I is the identity matrix [54] We can think of 0, and 0g as parameterizing diagonal and off diagonal terms respectively.
4.2 Loss landscape visualization
2075
-0405
-35
-13
-0.055
-15
-3.0
-85
04
1.0 -0.00
6;
(a)
(b)
Figure 5: (a) Weights of the teacher network. (b) Loss of the student network as we vary the diagonal and off diagonal weights of the + π3 map. Other weights of the student network are set equal to that of the teacher network. Red line corresponds to the fixed point subspace where 0z.
For n. block in the hidden layer, note there is hidden symmetry corresponding to permuting the corresponding n neurons. The resulting hidden fixed point subspace happens exactly the weights these neurons are all the same. If we have single T. block in the input, then the linear map 0,1 + 02(11" /) is in the hidden fixed point subspace when 0 02. In particular, the degrees of freedom is reduced exactly by Theorem 3.3 applies and we expect there be an additional induced minima. Note one can check that additional representations in the input or hidden layer do affect this.
Code for running the experiments is available at gei.hhcwwweeoniicccccccccaaeooee

---

Hence, we choose S3 and a 5-dimensional input consisting of one rep and 2 trivial reps. In the hidden layer we choose one सy rep and trivial To reps. We randomly generate weights for the teacher network. We then set all other weights of the student network to equal that of the teacher network and only vary the diagonal and off diagonal weights 01 and 02 controlling the mapping from R3 to z3- Figure shows the resulting loss landscape for random seed. We clearly see two distinct minima the loss landscape. In Appendix E we initialize teacher weights using different random seeds and see similar results.
4.3 Training, bad initializations, and parameterization
For each value of 01, 82, we trained the student network for 100 steps at learning rate of 10-1 using gradient descent the exact loss. This is usually sufficient for the student network to converge. The resulting final loss is shown in Figure 6a Note that indeed we see two regions, one which converges global minima and one which does However, the boundary
a straight line. turns this is because gradient descent behavior depends on parameterization and in particular, on the relative scaling of the parameters [31] By rescaling our parameterization vn 1 02 611" I) and then performing training, we obtain Figure 6b) Here, the boundary between the two regions is indeed straight line and corresponds the hidden fixed point space. We emphasize, however, that no matter how we rescale our parameters, there will always be boundary. In Appendix E we generate teacher weights using different random seeds and see similar results.
904
0.02
-05
-442
-05
-0.2
-14
-10-
-064
-15 +6.
-1.
+6.
(a)
(b)
Figure 6: Final loss achieved by student network after training for 100 steps. (a) Here parameterize the diagonal and off diagonal components directly. The boundary is curved in this case. (b) Here we rescale the diagonal component. We have straight line boundary in this case which corresponds to the hidden fixed point subspace.
4.4 Relaxation
Using the same setup as before, we randomly choose teacher network weights and choose a bad initialization for the student network. We train the student network until it converges the bad minima. Then, we relax the equivariance constraints completely and continue training until the network converges. The weights of the teacher network, bad minima found by the constrained network, and good minima found by the relaxed network are shown in Figure]
global minima
spurious minima
(c) relaxed minima
Figure 7: (a) Weights of the teacher network. (b) Spurious minima reached by equivariant student
network after training. (c) Global minima reached by unconstrained network initialized the spurious minima.

---

Note that the final weights found by the relaxed network are simply permuted with respect to the teacher weights. However, importantly this corresponds different group representation in the hidden layer. In order to traverse between these different choices, we must break equivariance. This process tracked in Figure g We initially have a negative Hessian eigenvalue, indicating that in the relaxed network the spurious minima becomes a saddle point. As we traverse between the two different equivariant subspaces, we see a bump in equivariance error. However, our loss still decreases. Finally, we arrive at global minima in different equivariant subspace and we return to no equivariance error. Note that the Hessian becomes positive definite indicating that we have indeed reached a minima not saddle.
Equnarencce
Min Hesskon Remivluue
0.00
-0.00
500
1000
1530
2000
2003
9000
Stegs
(a)
(b)
Figure 8: Training of unconstrained network initialized at bad minima. We break out of the original equivariant subspace into a symmetrically related equivariant subspace. In particular, the "bad" half of the original constrained subspace is closer to the "good" half of the symmetrically related cquivariant subspace. (a) Depicts values of various quantities during training of the relaxed model. (b) Projection of the parameters onto 3-dimensional subspace. The planes correspond to various symmetrically related equivariant subspaces color coded by good and bad halves containing the good or bad minimum respectively.
5 Conclusion
In this work, we presented a general framework for investigating loss landscapes of constrained networks. In particular, we proved that the parameter symmetries of the unconstrained networks leads to hidden fixed point spaces which can create critical points and even spurious minima in the con- strained networks. We then apply this to equivariant networks built with permutation representations, which can be viewed subspace of unconstrained MLPs. We characterize the hidden fixed point subspaces these networks. Finally, we experimentally demonstrate the existence of these spurious minima using teacher-studed setup. Further, we empirically find that removing the equivariance constraints can sometimes convert the spurious minima to a saddle allowing the network to escape. Interestingly, the final relaxed weights permuted version of the teacher weights and corresponds to a different choice of group representation in the hidden layer and hence a different equivariant subspace.
There are still a number of questions to be addressed in future works. First, our results require viewing a given model as a subspace of larger model class. Finding larger model classes which give useful insights is often challenging. In particular, it is not obvious to us how to so for equivariant models using tensor product operations. Next, Lemma 3.2 and Theorem 3.3 onlyy look at fixed point subspaces. However, these subspaces often have additional local parameter symmetries and we expect taking these into account give stronger, do not concretely understand when relaxation converts bad symmetry induced minima into saddle points. Finally, we believe that exploring relaxation techniques which change our choice of group representation could prove to be immensely helpful and fruitful area for future studies.
broadly applicable statement. In addition,
10

---

Acknowledgments and Disclosure of Funding
We acknowledge the support of the National Science Foundation under Cooperative Agreement PHY- 2019786 (The NSF Al Institute for Artificial Intelligence and Fundamental Interactions). YuQing Xie was also supported by the National Science Foundation Graduate Research Fellowship under Grant No. DGE-1745300
We would also like thank our helpful discussions with Robin Walters, Elyssa Hofgard, Hannah Lawrence, Vasco Portilheiro, and Yuxuan Chen.
References
[1] Emmanuel Abbe, Enric Boix Adsera, and Theodor Misiakiewicz. Sgd learning on neural net- works: leap complexity and saddle-to-saddle dynamics. In The Thirty Sixth Annual Conference on Learning Theory, pages 2552-2623. PMLR, 2023.
[2] Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. re re-bain:: Merging models modulo permutation symmetries. arXiv preprint arXiv:2209.0453 04336, 2022.
[3] Yossi Arjevani and Michael Field. Analytic characterization of the hessian in shallow relu models: A tale of symmetry. Advances in Neural Information Processing Systems, 33:5441-
5452, 2020.
[4] Yossi Arjevani and Michael Field. Analytic study of families of spurious minima in two-layer relu neural networks: tale of symmetry ii. Advances in Neural Information Processing Systems, 11155122-51772 2021.
[5] M. Artin. Algebra. Pearson Education, 2011.
[6] llyes Batatia, David Peter Kovacs, Gregor N. C. Simm, Christoph Ortner, and Gabor Csanyi. MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fields. Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[7] Simon Batzner, Albert Musaclian, Lixin Sun, Mario Geiger, Jonathan P. Mailoa, Mordechai Kornbluth, Nicola Molinari, Tess E. Smidt, and Boris Kozinsky. E(3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials. 13, May 2022.
[8] Arash Behboodi, Gabriele Cesa, and Taco S Cohen. A pac-bayesian generalization bound for equivariant networks. Advances Neural Information Processing Systems, 35:5654 -668, 2022.
[9] Alexander Bogatskiy, Brandon Anderson, Jan Offermann, Marwah Roussi, David Miller, and Risi Kondor. Lorentz group equivariant neural network for particle physics. In International Conference on Machine Learning, pages 992- 992-1002. PMLR, 2020.
[10] Denis Boyda, Gurtej Kanwar, Sébastien Racanière, Danilo Jimenez Rezende, Michael S Albergo, Kyle Cranmer, Daniel C Hackett, and Phiala E Shanahan. Sampling using su (n) gauge equivariant flows. Physical Review D, 103(7):074504, 2021.
[11] Johanni Brea, Berfin Simsek, Bernd Illing, and Wulfram Gerstner. Weight-space symmetry in deep networks gives rise permutation saddles, connected by equal-loss valleys across the loss landscape. arXiv preprint arXiv: 100i::1900.000 909909991,,... 2019.
[12] Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning, pages 2990-2999. PMLR, 2016.
[13] Taco S Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical cnns. arXiv preprint 1rXiv:1801.1011 2018
[14] Taco Cohen, Mario Geiger, and Maurice Weiler. A general theory of equivariant cnns on homogeneous spaces. Advances in neural information processing systems, 32, 2019.
[15] Yaim Cooper. The loss landscape of overparameterized neural networks. arXiv preprint
Stfi:18044102000 2018.
[16] Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non- convex optimization. Advances in neural information processing systems, 27, 2014.
11

---

[17] Mildred S Dresselhaus, Gene Dresselhaus, and Ado Jorio. Group theory: application to the physies of condensed matter. Springer Science & Business Media, 2007.
[18] David Steven Dummit, Richard M Foote, al. Abstract algebra, volume 3. Wiley Hoboken, 2004.
[19]
Bryn Elesedy and Sheheryar Zaidi. Provably strict generalisation benefit for equivariant models. In International conference on machine learning, pages 2959-2969. PMLR, 2021.
[20] Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. arXiv preprint arXiv:2110.06296. 2021.
[21] Mare Finzi, Max Welling, and Andrew Gordon Wilson. A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups. In International conference on machine learning. pages 3318-3328. PMLR, 2021.
[22] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference Machine Leurning,
pages 3259-3269. PMLR, 2020.
[23] Xiang Fu, Andrew Rosen, Kyle Bystrom, Rui Wang, Albert Musaelian, Boris Kozinsky, Tess Smidt, and Tommi Jaakkola. A recipe for charge density prediction, 2024.
[24]
Kenji Fukumizu and Shun-ichi Amari. Local minima and plateaus hierarchical structures of multilayer perceptrons. Neural networks, 13(3):317-327. 2000.
[25] Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems, 31, 2018.
[26] Mario Geiger and Tess Smidt. 200 2022.
e3nn: Euclidean neural networks. arXiv preprint
[27] Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation 3d, 2022.
[28] Arthur Jacot, François Ged, Berfin Şimşek, Clément Hongler, and Franck Gabriel. Saddle-to- saddle dynamics in deep linear networks: Small initialization training, symmetry, and sparsity. arXiv preprint arXiv:2106.12 15933, 2021.
[29]
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew J. Ballard, Andrew Cowie, Bernardino Romera-Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis. Highly accurate protein structure predic- tion with alphafold. Nature, 596(7873):583- 589,, 202.1
[30] Risi Kondor, Zhen Lin, and Shubhendu Trivedi. Clebsch- goodan nets: fully fourier space spherical convolutional neural network. Advances Neural Information Processing Systems, 31, 2018.
[31] Agustinus Kristiadi, Felix Dangel, and Philipp Hennig. The geometry of neural nets' parameter spaces under reparametrization. Advances in Neural Information Processing Systems, 36: 17669- 17688, 2023.
[32] Hans Kurzweil and Bernd Stellmacher. The theory finite groups: an introduction, volume 1. Springer, 2004.
[33] Jae Hyeon Lee, Payman Yadollahpour, Andrew Watkins, Nathan C. Frey, Andrew Leaver- Fay, Stephen Ra, Kyunghyun Cho, Vladimir Gligorijevic, Aviv Regev, and Richard Bonneau. Equifold: Protein structure prediction with novel coarse-grained structure representation. bioRxiv, 2022.
[34] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018.
[35] Yi-Lun Liao and Tess Smidt. Equiformer: Equivariant graph attention transformer for 3d atomistic graphs. In The Eleventh International Conference on Learning Representations, 2023.
12

---

[36]
Clare Lyle, Mark van der Wilk, Marta Kwiatkowska, Yarin Gal, and Benjamin Bloem-Redde On the benefits of invariance in neural networks. arXiv preprint arXiv:2005.00178, 2020.
[37] Albert Musaelian, Simon Batzner, Anders Johansson, Lixin Sun, Cameron J. Owen, Mordechai Kornbluth, and Boris Kozinsky. Learning local equivariant representations for large-scale atomistic dynamics. Nature Communications, 14(1):579, 20233
[38] Oskar Nordenfors, Fredrik Ohlsson, and Axel Flinth. Optimization dynamics of equivariant and augmented neural networks. arXiv preprint 1.XI::2303.135555 2023.
[39] Marco Pacini, Xiaowen Dong, Bruno Lepri, and Gabriele Santin. A characterization theorem for equivariant networks with point-wise activations. In The Twelfth International Conference on Learning Representations, 2024.
[40] Richard S Palais. The principle of symmetric criticality. Communications in Matherattical Physics, 69(1)19-30, 1979.
[41] Stefanos Pertigkiozoglou, Evangelos Chatzipantazis, Shubhendu Trivedi, and Kostas Daniliidis. Improving equivariant model training via constraint relaxation. Advances in Neural Information Processing Systems, 37:83497-83522 2024.
[42] Scott Pesme and Nicolas Flammarion. Saddle-to-saddle dynamics in diagonal linear networks. Advances in Neural Information Processing Systems, 36:7475-7505, 2023.
[43] Mircea Petrache and Shubhendu Trivedi. proximate) group equivariance. Advances in Neural Information Processing Systems, 36:61936- 61959, 2023.
generalizainnnnnnnnnn trade-offs under (ap-
[44] David Saad and Sara Solla. Dynamics of on-line gradient descent learning for multilayer neural networks. Advances in neural information processing systems, 8, 1995.
[45] Itay Safran and Ohad Shamir. Spurious local minima are common in two-layer relu neural networks. In International conference on machine learning. pages 4433-444) PMLR, 2018.
[46] Berfin Simsek, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, and Johanni Brea. Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances. In International Conference on Machine Learning. pages 9722-9732. PMLR, 2021.
[47] Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation- and translation-cquivariinn neural networks for 3d point clouds, 2018.
[48] Yuandong Tian. An analytical formula of population gradient for two-layered relu network and its applications in convergence and critical point analysis. In International conference on machine learning, pages 3404-344 3413. PMLR, 2017.
[49] Tycho van der Ouderaa, David W Romero, and Mark van der Wilk. Relaxing equivariance constraints with non-stationary continuous filters. Advances in Neural Information Processing Systems, 35:33818 1888888-3830, 2022.
[50] Dian Wang, Xupeng Zhu, Jung Yeon Park, Mingxi Jia, Guanang Su, Robert Platt, and Robin Walters. A general theory of correct, incorrect, and extrinsic equivariance. Advances in Neural Information Processing Systems, 56:40006-400299 2023.
[51] Rui Wang. Elyssa Hofgard, Han Gao, Robin Walters, and Tess Smidt. Discovering symmetry breaking in physical systems with relaxed group convolution. In Forty-first International Conference on Machine Learning.
[52] Rui Wang, Robin Walters, and Rose Yu. Approximately equivariant networks for imperfectly symmetric dynamics. In International Conference on Machine Learning, pages 23078-2309 PMLR, 2022.
[53] Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data, 2018
[54] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
13

---

A
Notation
For convenience, we provide table of some of the commonly used symbols in our paper and their typical meaning.
Table 1: Commonly used symbols
G
Group network is equivariant under
X
Typically used to represent the space of our inputs Typically used to represent the space of our outputs A group representation G -> GL(
Y
Input
Output
θ
Weight space of unconstrained networks
Se
Function Sa X + Y parameterized by E θ
P
Group of parameter symmetries
θ Linear subspace of parameters corresponding to enforeing some constraint
P Parameter symmetries of constrained class of networks, in particular we must have p E θ for p P
Н Hidden parameter symmetry group. H P and H L P which arises because our choice of constraint enforcement breaks the unconstrained parameter symmetries
и Fixed point subspace, we have hōи Oμ for all 0 O and g H
Hidden fixed point subspace, equal O O θH
T Identity matrix
Constant matrix of 1s Permutation representation.
B Other related works
Previously the optimization dynamics of equivariant, non-equivariant, and non-equivariant augmented networks studied in [88. This work showed that the sets of critical points of equivariant network remain critical points when we expand to an unconstrained space if we also include data augmentation. particular, under full augmentation, the chosen equivariant subspace is invariant under gradient flow. Our work is complementary in that we show how critical points can arise in the equivariant subspace due interactions with parameter space symmetries. In addition, our experiments show network leaving an equivariant subspace. This is consistent with [3] since we trained using SGD where our individual batches are not fully augmented. This means we can have gradient updates which leave the equivariant subspace.
Another very interesting series of works studies families of minima in 2-layer teacher-student ReLU setups 3 4 This is a similar setup to that used in our toy experiments with a model the form
f(;; W1, W2) W20 (WWa)
where Wi, W2 are trainable weights and a is pointwise nonlinearity chosen to be ReLU. In our setup, we fix W2 to be constant matrix consisting only of I's. In 4. they choose a teacher network such that W spectrum for the loss of the student network can be analyzed by exploiting symmetrics network from which they provide characterization of various families of minima. It turns out this analysis has intimate connections with our main theorems and the results [38] We briefly provide sketch of this connection below.
and W2 consists of all I's. The corresponding critical points and Hessian the teacher
In particular looks W1 isotropic under ASmm x AS,-m: consisting of simultaneously permuting the first m rows and columns, or simultancously permuting the last n = m rows and columns. It turns out such W, correspond exactly those satisfying equivariance constraints for = S,, 5n-m where we choose n.. no and x Rn n representations for the first m and next
neurons in the hidden layer. The minima families in 4 correspond exactly to those implied by our Theorem 3.3 if constrained to these symmetric Wi; specifically weights which have blocks where diagonal values
14

---

are less than the off diagonal ones. Because the input distribution is unit Gaussian and hence also symmetric under G, the results of [38
implies that even if relax the constraints on W,, the induced minima implied by our Theorem 3.3 remain critical points. Importantly however, this does not show they are minima in the unconstrained space. Only through the more detailed analysis of do we know that these are minima for wide enough inputs.
C Group theory background
Here, we present brief overview of concepts from group theory. We refer to standard textbooks for more comprehensive treatment S
17
18 32
We begin by defining what group is.
Definition C.1 (Group). Let G be nonempty set equipped with a binary operator : G G G. This is group if the following group axioms are satsfied
1. Associativity: For all a, b, G, we have (a b) cb- (b.ec)
2. Identity element: There an element E G such that for all G we have e.g g. =9
3. Inverse element: For all y G, there is an inverse g 1 E G' such that g y g for identity e.
Examples of groups include the group of permutations, the group rotation matrices with matrix multiplication as the group operation, and the group of integers under addition. One very important group is the group of automorphisms on vector space. This group is denoted GV((V and we can think of as the group of invertible matrices.
We care about using groups to describe symmetries. The group elements abstractly represent the symmetry operations and the effect of these operations is what we call group action.
Definition C.2 (Group action). Let G be group and Q set. A group action is a function G x Q + such that a(e, x) z and a(g, o(h,, z)) = a(gh, x) for all g, h G and Ω.
One may want to relate two groups to each other. In particular, we would want a mapping which preserves the group structure. Such mapping is homomorphism.
Definition C.3 (Group homomorphism and isomorphism). Let G and H be groups A group homomorphism function f G H such that f(u v) f()) f'v) for all u, v G. A group homomorphism is an isomorphism if / is a bijection.
Usually, consider data which lives in some vector space. Further, linear actions vector spaces are represented by matrices. Hence, it is particular useful to relate arbitrary groups to groups consisting of matrices. Such a homomorphism together with the vector space the matrices act is group representation.
Definition C.4 (Group representation). Let G' be a group and V a vector space over field F. A group representation is a homomorphism G - GL((V taking elements of G to automorphisms of V.
Given any representation. there are often orthogonal subspaces which do not interact with each other. If this is the case, we can break representation down into smaller pieces by restricting these subspaces. It is useful to consider the representations which cannot be broken down. These are known as the irreducible representations (irreps) and often form the building blocks of more complex representations.
Definition C.5 (Irreducible representation). Let G be group, V vector space, and p: G GL(V) a representation. A representation is irreducible there is no nontrivial proper subspace W C V such that plw is a representation of G over space W.
There has been considerable work on characterizing the irreps of various groups and many equivariant neural network designs use this knowledge.
Another special type of group representation a permutation representation. It turns out only these types of representations are compatible with arbitrary pointwise nonlinearities [39.
Definition C.6 (Permutation representation). Let G be group. Any group homomorphism π G ->> (n) is abstract permutation representation. Let p: Sym(n) + GL(V,) be the representation
15

---

mapping to corresponding permutation matrices. Then pon: G - GL(V,) is permutation representation. We will abbreviate permutation representation as p-rep.
The intuition is that for a p-rep, there is always some choice of basis b, such that p(g)b; (x In other words, group actions just permute the basis. The trivial representation is an example since there is only one basis and it remains invariant under group action. The regular representation commonly used group convolutions is another example since we can associate each basis vector to a group element 12
Just like how irreps are building blocks arbitrary representations, we are interested in the building blocks of p-reps. Such p-reps the transitive p-reps. For such p-reps, each pair of basis vectors can be transformed into each other through a group action.
Definition C.7 (Transitive permutation representation). Let G group and p + GL(Vw) be permutation representation. We say the permutation representation is transitive if there is basis b,,...,b Dn of Vn such that the action G on the set (bi....,6, bn)) is transitive.
It turns transitive p-reps also be characterized and it is much simpler than for irreps. To do so, we must introduce some additional concepts.
First, we introduce subgroups. A natural question is whether subset of group elements themselves form a group under the same group operation. Such a subset is a called a subgroup.
Definition C.8 (Subgroup). Let be a group and C G. S together with the group operation of G. satisfy the group axioms, then S is subgroup of G' which we denote as S < G.
One particular feature of a subgroup is that we can use them to decompose our group into disjoint chunks called cosets.
Definition C.9 (Cosets). Let G be group and S subgroup. The left cosets sets obtained by multiplying with some fixed element of G on the left. That is, the left cosets are for all E G
g$ {gs: s S).
We denote the set of left cosets as G/S. The right cosets are defined similarly except we multiply with fixed element of G' on the right. That is, the right cosets for all G
Sg {sg S..
We denote the set of right cosets G.S..
Next, it is useful to define what we mean by symmetry of an object. These are all group elements which leave the object unchanged and is called the stabilizer.
Definition C.10 (Stabilizer). Let G be a group, Ω some set with an action of defined on it, and E Ω. The stabilizer of u is all elements of G which leave u invariant. That is
Stabe (u) = [g gu = u E G}.
One can check that the stabilizer is indeed a subgroup. Closely related to the stabilizer is the orbit. This is all the values get when act with our group some object.
Definition C.11 (Orbit). Let G be group, Ω some set with action of G' defined on it, and u E Ω. The orbit is the set all values obtained when we act with all elements of G on it. That is,
Orbe(u) = {gu E G} = Gu.
It turns one show that the stabilizer of elements in the orbit related. This relation turns to be conjugation which we define below.
Definition C.12 (Conjugate subgroups). Let G be group and S and S' be subgroups. We say S and S' are conjugate if there some g G such that
gSg =
Conjugacy of subgroups defines an equivalence relation.
16

---

Definition C.13 (Conjugate classes of subgroups). Let G be a group and S a subgroup. Denote the conjugate subgroups of the
Clc(s)= = l G).
The conjugate classes of subgroups are defined as the different sets of conjugate subgroups.
Remark C.14. The conjugate classes of subgroups for finite subgroups of O(3) correspond to the point groups. This is the reason why certain groups such as D3 and C3. are named differently despite being isomorphic. No D3 subgroup can be transformed into C3v subgroup via conjugation by some element of O(3).
Now we can characterize the transitive permutation representations.
Lemma 2.6 Let G be group. There is a canonical bijection between the transitive permutation representations of G and the classes of conjugate subgroups.
Proof. First, given any conjugate class of subgroups C, we can always pick subgroup S C. Then consider the cosets G/S and pick coset representatives [.),.... J. We can construct basis and define the action of g on by as gb; = b, where j is such that fj = 99:S. Hence, we can map each conjugate class of subgroups to a transitive permutation representation.
in b,,...
Let p: G GL(V)) be transitive permutation representation. Suppose bj...., b, is basis of V, where P consists of permutation matrices. Then consider the mapping P to Clg (Staba(b1)) Note easy to check that this inverts our previous construction of a t-rep from a conjugate class.
Now consider some different transitive permutation representation p' G + GL(V,) where By..... 8/2 is a basis where o' consists of permutation matrices. Further, suppose Cla (B Cla (Stabaa)) ))
Then there must be some G such that (Stabe:(h)) = Stabo(t()) Then define mapping given by x =g)b() = g gbi for g' E G. Note that (tab = gStabg (b, )g Stabe which means this map is consistent. Note express change of basis matrix B. But then
B p(""("")
Bplg")) gb B- (g" g'd"""))bb = g' = p D'
Hence, P and p' equivalent representations. Lastly, we state Schur's lemma which a very useful tool for characterizing equivariant linear maps. Theorem C.15 (Schur's Lemma). Let G be group and V, W be 2 vector spaces equipped with irreps PV, PW-
1. If Pw are not isomorphic, then there are no nonzero G-equivariant maps V + W
2. If pv: Pw are finite dimensional over an algebraically closed field and pv = pw, then the only notrivial G-equivariant linear maps V > W scalar multiples of the identity.
C.1 Infinite permutation representations
Permutation representations are typically defined the finite case. Here we show how this definition can be extended for infinite dimensional representations.
First, recall that the abstract permutation representation was defined homomorphism G Sym(n). We can view Sym(n) as Aut(X) for some set of cardinality |X| = n. We can instead replace the finite X with some arbitrary (possibly infinite set).
Definition C.16. Let X some set. Define homomorphism x G - Aut(X) as a generalized abstract permutation representation.
We can then view any function f X F where some field as an infinite dimensional vector space. In particular, we can consider some Jx with some canonical basis e, so that any e E T can be written as
Σ f(x)cz.
ZEX
Using the generalized abstract p-rep n, can define action on this space
17

---

Definition C.17. Let G be group, X be some space, be a field, and x G + X generalized abstract p-rep. Then we can define map p: G - GL(FF) by
pl))== p()
Σ =EX
(a)e
(T
=ex
=EX
where e - -2Ex (x)cc E Bxx is any vector. This is what we call generalized permutation representation.
An example of generalized p-rep is S signals. If we embed S as a unit sphere in RR, then the standard rotation action R. on R$ gives natural generalized abstract p-rep. In particular, we can simply define π (O(() + Aut(S²) by
T(R)x Rx.
We simply just rotate the sphere.
Hence by Definition C.17 for any spherical signal f S² + R, the action p(g) f(R defines a generalized p-rep the infinite dimensional space of spherical signals.
D Proofs
Lemma 3.2 (Hidden symmetry induced critical points), Let Fe C Fe be a class of constrained networks. Let L θ + R be loss associated to these networks. Let H be hidden symmetry with orthogonal action and let Oн {0 θ h0 00'0 vh. H) and O θ n.
If (O (o) + ( n θ) = θ, then critical points of Lleg are also critical points of L
Proof. We first note that any linear invertible action on θ induces an action on the tangent space. M E GL(@) is invertible matrix, then this induces action function f e + R given by f(0) - (0) = f(M-10). . By chain rule, we find that
(V)' = (0) = 3.r(M 0) 0g MRJ 0, = (VJ(M- M))
where we used Einstein summation conventions. Viewing V as row vector we get
f (M~
so there is an additional action of (M-1)T in the tangent space.
Because T. is symmetric under H, we have
(((0) (h- (LL(O)))() (h (1-1)TV(L(0))
Since assume H has orthogonal action, then h- = 1tT
VL(0) (h-1) -1-1)" LV
By definition of 1h, this means must have (())) E OH for any E O
Suppose 0* is critical point of Leg particular, for 0 have VL(0") E Or. For any direction t E Ô, we can uniquely decompose
t tH + f for t θh and tu E 0/ But since é (0 n é) (O θ), must have tu (0 )) and t (0) n). Therefore,
VL(00) =(1 + (1)
=t =0
since showed VL(O") Og and assumed 0* is critical point of θH.
18

---

+ 66 2 Lemma D.1 (Minima on boundary). Let U be an open subset IR". Suppose f f + - C' smooth function. Consider a point a aU on the boundary which locally C smooth so that we can define outward pointing unit normal f(x)). Then if v(x)Vf(()) > 0 then cannot be the global minima of f.
Proof. Let 8 - - (z)). There exists some such that for all where (u. - -x| < we have [f(u) [(x)) - (u x) Vs(z) < df In particular, suppose we choose u = x ((()). Then
2-
[(u) <f(x) + (u x) vf(x) + 2
= =f(x)
6 ((x)) Vf(x)
2
8 + cd
$ f(x)
2
2
(x))
Therefore x cannot be the global minima.
Lemma D.2 (Existence of minima in functions on balls). Let B be a closed n-dimensional ball. Suppose f B" + R is C¹ smooth function. Suppose for any point on the boundary ÖB" we have fxx) V fx) > 0 where ñ is the normal vector out of the boundary. Then f has an absolute minima in the interior B".
Proof. First, because f is a continuous function over a compact space, it has an absolute minima by the extreme value theorem. Further, since the boundary C² smooth and f(x)) ffx) > Lemma tells us there cannot be minima on the boundary. Hence we conclude the global minima of f lies the interior 1
Theorem 3.3 (Hidden symmetry induced minima). Let F&g Fe be class of constrained networks. Let L. e R. be loss associated to these networks. Let H be a hidden symmetry with orthogonal action and let θH = (0 θ hd = vh E H) and θμ = O n θ.
Suppose ( n 6) + (O n Q) = ê and dim( (Ôn) = dim - 1. Suppose there is a minima 01 of Ll such that 01 & θн. Further suppose that there exists some C' such that for any direction i E θ we have f. VT(CF) > 0. Then if Hessians of all critical points nondegenerate, there must
exist a distinct minima 02 + 0g of Lla-
Proof. First, consider the closed ball Bc (0 θH 10 boundary we have f. VC(CF) > O so Lemma D.I tells us has a minima 0 in the interior (0).
C}. By assumption, on the
So 8* must be critical point and in fact must be a local minima of L But by Lemma 3.2 0" is also critical point of Lle-
If 0* is a local minima of L/g, then we have additional minima distinct from 0 since 02 θ but l" Ô.
So suppose 8* is not a local minima of L Let E θ be a unit vector normal to θg. Then we can consider two hemispheres given by
D {i E θ : á < 0, |x| S C)
D2 fx θ : 2 0, |x| S C
We claim neither of these sets can have minima on their boundaries. Without loss of generality consider Dj. Its boundary consists of S; = E 1 : 0, |>| = C) and Bc Along S1, by assumption ni(:)- V L(x)) 0 so Lemma D.2 tells the global minima of D2 cannot be on Sj. On (O), we know the minimum value is L(d).. However, we assumed 6* is not a minima so must be saddle. Since we assumed nondegenerate Hessians critical points, there must be some unit vector d such that d' H (( < O. Note that since 6* minima of Le |d n O. Without loss of generality suppose d ft < 0.
19

---

Then for any ó there exists some such that for all 0 E θ, if 10 - 0- c then ((00)-(0)) - 2 (0--
8-)" (( (B 8*)l 10 88 Pick 5 H|dd H())) )d and 0" +cd/2. Since d. we
see 0. so Dj. But we see that
I(0) <I(0") + ed
ed + 4
H(0))
2 2
2
=((") 1 (-6)
2²
2
4
= L(0")
4 e
<I(0")-
Therefore there is point in the interior of D1 with value smaller than L(0*). Hence the minima also cannot be on (0). Hence D,D, must have its global minima in the interior. Similarly L D. must well so there are 2 distinct local minima.
Theorem 3.4, Let m-neuron point be transitive permutation representation. Suppose the weights satisfy the corresponding equivariance constraints. If this m-neuron point is reducible, then either we can eliminate these neurons we can replace the transitive permutation representation with n-neuron transitive permutation representation where m and n m.
Proof. Let W be the input weights with dimensions X dm and U be the output weights with dimensions dout X m. Since the hidden neurons transform as a permutation representation, for any g E G denote by (t) the node which g transforms node i into. Denote by p, p', p" the representations of G on the current m-neurons (R)",, previous layer (IRdin) and next layer (Reoau) respectively.
There are 2 cases for reducibility. Either we have a neuron with 0 output weight or pair of neurons which share input weights.
The first case is that there is some & where Us = for all i [doul] In this case we find that by equivariance constraints for any g E G we have
Uin, (k) (U p(g)) = (g) 0.
Therefore all output weights are O and we can eliminate this transitive block of neurons.
The second case is that there exist distinct k k such that Wa. = WR, for all i E {1, (1.2.... .2...,d,.. di Suppose h E G such that T4 (k') = k. Then by equivariance constraints we must have
(p(h))))s: A+ W (1)) Wes (Wp'(h))&z (h)
where we use Einstein summation notation. But since Wky Wki we have
We. Wa)p/,(h)
for all such h. Further, let S₁. be the stabilizer of & under æ. That have (k-1(k) for all E S
and E (1,2, J. Then by equivariance constraints we also have
(p(s)W) K4 = We: = (K)
(2)
Now, consider group generated by elements of S₂ and h. Then any A can be written
91.2 9p
for some finite p where each di is either in S₁ or h. We then find that
Waspsx(a)
= Phij (91) p (WKj P (91)) ... 1 (g)) WKizPhart (92) (1))
Wh
20

---

where we repeatedly apply and 2. However, by equivariance we must also have
Ws: (a) (p(a)W) Wx, 18)s
for all E .I, and any E A.
Next, consider some coset gA E G/A. We then find for any b ga g we have
W W.,(k): =((bb ((o((b'))sss W)) Waspfe(b)))
-Wispsila 1)
=Wanifjisaa pa='))/as00)))
-WajsPizl00
Hence, that for any g from the same coset, the incoming weights (k); the same. Since our permutation representation is transitive, all neurons are reached by appropriate choice of g E G. Hence, we can reduce our m-neuron point to
point.
Finally, we must show that with this reduction we still satisfy equivariance constraints. To do so, let us define new weights and new representation. First, pick some coset representatives {gi = e, 92. J. Next, we define new ingoing weights
W
W Top (4)9
and outgoing weights
U¹
Σ (k))(k) gEgrA
Let p: G + GI(RIG/AI) be permutation matrices defined as
م:g(9) = {
99; E 9
otherwise
One can check that these matrices indeed form a permutation representation, Further, denote by Ag [|G/A|] + [|G/A]] be the corresponding permutation of indices for p(g)-
We then check for any g E G that
(W' p''''(()) (2)))m -W papli (4) = W mop (k)spPy (g)
=pksl97 Wt PkJ (9p (p))))
=Pkelop g)W = W5 - (b)g
Now let By be such that g 9p g,A. Then we see
(W' (')'())))r W no,((000 (h)a = W'4
Pyr W9)
=(p())W')Pr
So W' satisfies the equivariance constraints.
For U we check that
(g"()) Po)(
= Σ
(1)
I A
Σ UpiPse,,10))
8' g'EgpA
Σ
UniPes (1)pp(()))))
g'EgpA
Σ
UpsPaa(ag")
o' g'Ea2A
21

---

Let 9j be such that 99p E 9jA. Then for any g' Ap we have g' Ipa for some A so gg' -9pa gy a since we know 99p = gya' for some d'. So then gg' E gj A. Hence we see that
(g'(g))))
E UgiPalg")
8"E9,
- = (0)
-(U'((9)) (()))
So the new output weights also satisfy equivariance constraints.
Theorem 3.5 Let m-neuron point and m'-neuron point be distinct transitive permutation representations each of which are irreducible. Suppose the m + m' m'' nneurn point combining these neurons is reducible. Then they must both be the same permutation representation and we can replace the m + nneuon point with a m-neuron point.
Proof. Let use denote the incoming weights the m-neuron and n'-neuron points as W and W respectively. Suppose k, are such that
Wk: We
for all E [dn]-
First, we show that both the m and m' neuron points are the same permutation representation. Let
S = (k)
S' Stabb(k)).
Suppose S + S' Then either there is some E S where & S' or there is some s' E S but s' & S. Without loss of generality suppose we have the former. Then by equivariance, we have
W1y =(p(s =(o(s-))
(Wpls )))s [WkiPty W ')
(W"p(s> (o'(=-))W") Waay
But since $ S', (k') * W. But this would mean m' -neuron point is reducible, a contradiction. Hence we must have S'. Because both neuron points are transitive, this implies they must be the same.
To conclude, note that equivariance constraints uniquely define the remaining rows of W, W which means there is a bijection between rows of W, W where all corresponding rows are equivalent. Hence, we can combine all these neurons can create single m-neuron point instead.
E
Additional experiments
E.1
Permutation symmetry
Using the same setup as in Section 4 weights.
we tried different random seeds for initializing the teacher
E.1.1 Loss landscape
we have the loss landscape as we vary the diagonal and off diagonal components of the + map for various random seeds. There seem to be two distinct minima in most of the plots. For Figures 9b] 9i] [9]] the distinct minima not very apparent. By directly checking the weights, we were able to confirm that the diagonal and off diagonal components of the teacher weights are very close which likely led this phenomenon. In addition, for Figures Sngures@eeann and Dg] the minima are out of range of our parameter sweep. Interestingly, we note that the global and spurious minima always appear to be roughly equidistant from the diagonal line corresponding to the hidden fixed point subspace.
In Figure y
22

---

-03
(a)
(b)
(c)
(d)
Cite
GIM
600
400
(I)
(g)
(h)
ii)
Figure 9: Loss landscapes for various random seeds. Note there seems to be two distinct minima in almost all cases.
E.1.2 Constrained training
Same before, we initialize student networks have the same weights the teacher network except for the diagonal and off diagonal components of the т3 + map. We then train for 100 steps with gradient descent and record the final loss. We do this both for direct parameterization of diagonal and off diagonal values (Figure an] one with rescaled diagonal values (Figure IT) As before, we see clear boundaries in most of the cases and in Figure T0 we generally see nonlinear boundary while in Figure 11 we generally see a linear one which corresponds to the hidden fixed point space. However, for (b) and (i) there seems to a different boundary which much more prominent. is unclear to exactly where this boundary comes from. We suspect that there may be additional spurious minima not explained by our symmetry argument. Such minima are known to be common
in these 2-layer ReLU setups 45
600
66
-000
-00
422
(a)
(b)
(c)
(d)
(c)
600
Cooa
606
2200 44
(I)
(g)
(b)
Figure 10: Loss after training student network for 100 steps. We directly parameterize the .3 T3
map as 0,1 I + 0,1
1).
E.2 Different activation
Finally, we also tried replacing the ReLU activation with an erf activation. This was chosen because there an analytical loss for teacher-student setup for iid unit Gaussian inputs 44). The results are shown in Figure [2 We also tend to see additional minima expected from theory.
23

---

600
61
13
(b)
(d)
60
100
(I)
(h)
ii)
(i)
Figure 11: Loss after training student network for 100 steps. vn I0 / + 02 021 -)).
rescale the m3 +
T3 map as
65
(a)
(b)
(c)
(d)
A35
30
03
+15
-10 -0
(f)
(g)
(h)
(i)
Figure 12: Loss landscapes for various random seeds. Note there seems to be two distinct minima in almost all cases.
E.3 S2 signals
It turns out spherical signals are also permutation representations. This is result of (0(3)SS0 s S² Spherical signals are used 5 activations first proposed [3] many equivariant archi- tectures, spherical signals are parameterized in the Fourier basis using spherical harmonics [6] 47). Next, we note that any automorphism of S2 in principle gives hidden symmetry and the corre- sponding fixed point subspace is constant signal. This corresponds to only scalar 0 spherical harmonic. Hence, our inputs only consists of scalar
spherical harmonic and a single nonscalar spherical harmonic, Theorem applies and we expect multiple minima.
To test, this, we construct simple model. First, we pick some cutoff Lmax for maximum degree of spherical harmonic used. We then pick some representation for our input by specifying the relevant SO(3) irreps. We construct an equivariant linear map from our input representation to 0 @ + Lmmax used to represent the 5 signal. We then apply our nonlinearity the sphere and then extract the harmonics of the final signal. Finally we dot product final harmonics with itself and sum obtain output. We the e3nn library to build this model (26).

---

E.3.1 Single scalar and nonscalar irrep
We first consider the case where input rep is of the form 0 L. This is the where Theorem 3.3 applies. For such a network, by Schur's 's lemma there are only two degrees of freedom in the equivariant linear layer. We map the and L irreps the 0 and L harmonics of the spherical signal. Mathematically, we have the signal
f(f) = 6oz (0) + +0g
Σ m
Zm y
where r0()) is the scalar input and Jm is the irrep input and 0o, 01 are the weights.
(2)
The hidden fixed point space corresponding
automorphisms of the sphere is precisely constant signals on the sphere. This happens when 02 0. Hence by Theorem 3.3 we expect there to be two minima corresponding to positive and negative values of 0₁. We randomly choose values of Oo, O for the teacher network then sweep over Oo 0, and compute the loss of the student network. The resulting loss landscape is shown in Figure
<
(a) 0 +
(b) 0
(c) 0
(d) 0 @ 4
Figure 13: Loss landscapes for various random seeds and different Again, note the existence of two minima. However, the landscape appears symmetric for odd L but not even L.
Note that the loss landscape appears symmetric for odd values of L. It turns out that this is because the inversion map on 2 gives simple sign flip on odd spherical harmonic values. Hence this is parameter symmetry of such model and our previous hidden fixed point space is in fact just fixed point space for odd L. So for odd L we expect the two minima be the same. However, for even L, the 0 () line is truly a hidden fixed point subspace and we expect different minima.
E.3.2 Other input reps
We also tried testing other combinations of input reps for which Theorem 3.3 does not apply. In particular, we consider inputs of type L1 + L2 such that for (I1) and (122 input we obtain signal
f(f) = 01 Σ
(L)) y
Σ 2 a a(la)y
where 0,, O2 are the weights. We again randomly initialize these weights the teacher network and sweep over 0,, Og for the student network and compute the loss. Interestingly, we still often observe multiple minima reminiscent of the patterns expected from our other experiments.
F
Loss landscape and different optimizers
We provide a brief discussion of how our loss landscape characterization affects our understanding of optimization. general, because Theorem 3.3 implies existence of multiple minima, we expect potential issues where gradient based optimizers converge to the wrong minimum. In addition, we note that for certain "nice" parameterization, gradients in the hidden fixed point space stay in that space. Hence gradient descent and SGD would never cross this space. If we model weight decay with I.2-regularization, we see that regularization term does not affect the symmetry the loss landscape so this barrier remains. However, for different parameterizations equivalently, methods assigning different learning rates to different parameters the hidden fixed point space could pose less of an issuc.
25

---

15
15
10
-55
-03
-05
-10
-10
-19
1.5 -1.0
-0.5
1.5
1.5
2.0
11.
-10
-65
2
(b) 2 @ 6
(c) ()406 6
Figure 14: Loss landscapes for various random seeds and different input types L1 G L2. Even though Theorem 3.3 does not apply, we still often observe multiple minima.
Lastly, Lemma 3.2 implies existence of critical points in a much more general setting. While these may be saddles instead of minima, saddles are still considered problem for many optimization methods [16]. Further, a number works have shown sadle-too-aadle dynamics under various settings [1 [2). Hence we believe Lemma 3.2 may have important consequences to be explored future work.
26

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificcton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We clearly summarize our main theoretical contributions and experimental evidence. We also highlight the key takeaways as bullet points.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
27

---

Justification: We briefly highlight limitations of our paper in the conclusion and how future works could improve upon
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Rigorous statements are provided the main text and proof's can be found in Appendix D
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The architecture and setup of our toy experiments are clearly described.
Guidelines:
28

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provide link to code which will be made public.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
wwt.eesubbissiooooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https //nips.cc/publi///guu
Codeeuumiiii wettsbbmmssiiioi ssionpolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
29

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We explain the setup of our toy experiments in the main paper. The full details are in the code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: We only have toy experiments meant to support our theoretical loss landscape insights. Statistical methods are not applicable.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
All of our experiments are simple and can be run a laptop.
Guidelines:
The answer NA means that the paper does not include experiments.
30

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpps///arrrp.....o.b.l///tcaaa
Answer: [Yes]
Justification: conforms.
We confirm that we have reviewed the code of ethics and that our work
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work is foundational and seeks to further our understanding of neural networks.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
31

---

Answer: [NA]
Justification:
Our work is mostly theoretical.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have cited the packages used in our toy experiments.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No assets released, work is theoretical.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
32

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing or human subjects were used.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: No crowdsourcing or human subjects were used.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs were only used to help edit the writing.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our L.MM policy huttps://neurd:: Cwwpurrppsscc/Coffeeeeeeeee / for what should or should be described.
33


Reviews:
[{"review_id": "N5yv2KK3dn", "reviewer": "Reviewer_cpcL", "summary": "The paper investigates the consequences for considering the loss-landscape of designing neural networks to be equivariant. This is done with an MLP as a case study. The authors adopt the view that equivariant networks should be seen as part of a larger family of un constrained networks. In this case it turns out that the loss landscape of the equivariant networks may have minima which are saddle points in the case of unconstrained networks; a feature which would lead to undesirable training behavior.\n\nIn general the idea is interesting and well founded. I thus recommend that the paper is accepted.", "questions": "- Line 24-25: I assume it is an error that this sentence is included?\n- Line 85: I suggest that the authors provide a reference to how maximally expressive equivariant linear layers are constructed forn an irrep decomposition.\n- Line 94: Is it assumed implicitly that G is finite?\n- Line 105: Isn't it meant to be $\\rho \\circ \\pi: G \\to GL(V_n)$ instead of $\\rho: G \\to GL(V_n)$ to be consistent with previous notation?\n- Line 107. I assume that the action of $G$ on the set $\\{b_, ..., b_n\\}$ is transitive if it only have one orbit?\n- Line 109: Please provide a reference or a proof for this Lemma.\n- Line 128: Why is it needed that $p$ is a homeomorphism (wrt. the Euclidean topology i assume)? Is it not sufficient to require it to be bijective for P to be a group?\n- Line 152: There seems to be something wrong with the sentence \"Then if $\\Sigma is ...$\".\n- Line 179: By linearly constrained subspace do you mean a fixed point subspace?\n- Line 180: Which group is $p$ an element of and what is the action?\n- As mentioned, a main point of the paper, as I understand it, is that equivariant networks might introduce minima, which would not appear in a corresponding unconstrained model. However do you have any reflections on whether this poses a problem empirically? I ask since often networks will be  extremely overparametrized, and we rarely seek a global minima anyway. As such one could think that the additional minima could be introduced through the use of an equivariant model would not pose a problem.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n- The idea is interesting and compelling. I applaud that the authors for writing a paper aimed at principled understanding of a topic.\n- The manuscript is in general well written.\n- The authors comprehensively reflect on the limitations of their method.\n\nWeaknesses:\n- The paper has a variety of central results (Theorem 3.3, 3.4 and 3.5). Unfortunately the proofs of all these important results are moved to the appendix. I would suggest to the authors that they provide either the full proof or a sketch of proof in the main body of the paper.\n- I am slightly confused about section 3.2 overall. As I understand it, it is supposed to characterize hidden fixed point spaces of equivariant networks. However, I do not see the direct connection to the previous sections.\n- I urge the authors to work on the clarity in communication of the ideas proposed. For instance a main point of the paper, as I understand it, is that equivariant networks might introduce minima, which would not appear in a corresponding unconstrained model. I suggest that the authors to attempt to make it even more clear how this happens and why it is a problem, e.g. through further use of illustrations or examples.", "was_revised": true, "final_justification": "The paper addresses an interesting and relevant problem, and the authors provide a principled approach to understanding the topic. The weaknesses I observed in the paper was mainly related to communication of some theoretical results. However, the authors were able to mitigate this weakness during the discussion and rebuttal phase.", "rebuttal": "We thank the reviewer for their insightful comments. We appreciate that the reviewer finds our work compelling, principled and well written. We hope our response addresses the weaknesses and questions raised.\n\n## Weaknesses:\n* We agree that it would be nice to include proof sketches but were unable to due to space constraints. Instead, we provided Figure 2 giving an intuitive argument for why Theorem 3.3 is reasonable and Figure 3 which tries to visually explain the results of Theorems 3.4 and 3.5. We felt this provided a reasonable balance especially since some of the proofs are rather mechanical. However, we are open to further suggestions!\n* Thanks for this feedback! The high level idea is that the fixed point subspace for a parameter symmetry of permuting two neurons corresponds to those two neurons having the same input and output weights. From Lemma 3.2 and Theorem 3.3, we care about how this fixed point space intersects with an equivariant subspace. What Theorems 3.4 and 3.5 show is that within an equivariant subspace, equivalent neurons cannot happen in isolation. Instead, the existence of a single pair of equivalent neurons must imply a number of other neurons also be equivalent in a structured way as depicted in Figure 3.\n* Thanks for this suggestion! If accepted, we will add figures to motivate/describe the hidden symmetry idea which underlies our main results. We hope the reviewer finds the existing Figure 2 insightful for motivating Theorem 3.3.\n\n     In addition, we tried to highlight the key takeaways in the current manuscript at the end of both the abstract and introduction. Upon reflection, we believe the following modifications for the first 2 points will make the ideas more clear.\n\n     * By viewing a given architecture as a constrained version of a larger function class, we prove that the previously hidden parameter symmetries of the larger class can imply the existence of critical points and even spurious minima for the constrained network\n     * Hidden symmetries induce multiple symmetrically related but different constrained subspaces. Empirically, we find a spurious minimum in an equivariantly constrained subspace is often a saddle in the unconstrained space and escapes to a better minimum in a symmetrically related subspace.\n\n## Questions:\n* Line 24-25: This is meant to be a parody of the famous first sentence of the Tale of Two Cities novel which inspired our title. This can be removed if the reviewer feels strongly.\n* Line 85: Good suggestion! We will add references to standard group theory texts for Schur’s lemma. In addition, we will add a discussion of Schur’s lemma in the group theory section of the appendix.\n* Line 94: It is not required that $G$ be finite, however the permutation representations are finite in this definition. In principle, we can extend the definition to cover infinite dimensional representations by replacing $\\mathrm{Sym}(n)$ with $\\mathrm{Aut}(X)$ for some object $X$ in Definition 2.4. We can add a discussion of this extension in the appendix.\n* Line 105: Yes, although we would like to only use $\\rho$ to simplify the writing. We will make sure to clarify this in the revised version.\n* Line 107: Yes, exactly!\n* Line 109: We did put this in the appendix (but forgot to fix the numbering). It starts on Line 545 or also Lemma A.15.\n* Line 128: You are right that bijection is all we need. We originally wanted to only consider “nice” transformations which is why we chose homeomorphisms.\n\n     However, upon closer inspection, we realize that we assumed linear transformations in our main theorems without mentioning it and will update our manuscript to include this. This is not a major restriction as most commonly considered parameter symmetries (permutations, scaling) are linear transformations.\n* Line 152: We believe changing this to “If $\\Sigma$ is …” should fix this\n* Line 179: No, we mean a subspace satisfying some other linear constraints (e.g. equivariance constraints)\n* Line 180: Thanks for pointing this out! We meant for $p$ to be a parameter symmetry of the unconstrained networks.\n* This is a good question! We do indeed expect the spurious minima problem to mostly disappear as we increase width. For wide networks, we expect there to be enough transitive blocks initialized on the “good” side so a network could learn to only use the “good” blocks and ignore the “bad” blocks. Further, if a previous layer is wide, the dimension condition of Theorem 3.3 is less likely to hold.\n\n     Finally, while the dimension condition of Theorem 3.3 restricts its applicability for wider networks, we emphasize that Lemma 3.2 has very mild conditions and will still imply hidden symmetry induced critical points! This is still very important because saddle points are a known problem for optimization [1]. Further, this Lemma lays the foundation for future work which may investigate how the number of non-global minimum critical points grows as we overparameterize, similar in spirit to [2]. This could potentially give insight on differences in scaling behavior between equivariant and nonequivariant models.\n\nWe thank you again for your time in reviewing our work. Please let us know if you have any other concerns.\n\n#References\n\n[1] Dauphin, Yann N., Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. \"Identifying and attacking the saddle point problem in high-dimensional non-convex optimization.\" Advances in neural information processing systems 27 (2014).\n\n[2] Simsek, Berfin, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, and Johanni Brea. \"Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances.\" In International Conference on Machine Learning, pp. 9722-9732. PMLR, 2021."}, {"review_id": "kF34pZjEHo", "reviewer": "Reviewer_Ppai", "summary": "This work investigates the question of whether equivariant constraints in neural networks can create optimization challenges. First, the authors provide an extensive presentation of the preliminary background that is necessary for their analysis. Then, they focus on the analysis of equivariant models that can be derived by constraining the parameters of a fully unconstrained model to a specific parameter subspace. They define the concept of hidden parameter symmetries, which are symmetries in the unconstrained space that disappear in the equivariant subspace, and they show how these can induce critical points or spurious minima in the loss landscape of the constrained equivariant model. Additionally, the authors construct a simple example of a single-layer model where these critical points can be observed experimentally, and they demonstrate how the parameter space can be split into two distinct halves that cannot be traversed by gradient flow. As a result of the theoretical analysis and simple experimental observations, the authors conclude that, indeed, partially removing the equivariant constraints can result in an easier optimization landscape, allowing the network to utilize different types of latent equivariant representations and traverse distinct equivariant parameter subspaces.", "questions": "- Can the results of this work be used to provide a guideline on how to relax the equivariant constraint of a given model?\n- How reasonable is it to expect that the separation of the parameter space into two halves can also happen in deeper models with a much higher number of parameters?\n- How do these critical points induced by hidden symmetries affect optimization if, instead of gradient flow, stochastic gradient descent is used?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- The topic of this work is timely since there are multiple works that showcase evidence supporting the benefits of symmetry breaking in equivariant neural networks. Additionally, while most previous works provide mostly experimental evidence, this paper presents a theoretical hypothesis on the possible mechanism behind this phenomenon.\n- The presentation of the theoretical analysis is clear and can be easily understood even by readers who might not be experts in the topic.\n- The simple experimental setup provides both evidence supporting the theoretical hypothesis and also helps the readers build a better intuition on the problem at hand.\n\nWeaknesses:\n- While the authors provide a possible mechanism showcasing how an equivariant constraint can create optimization difficulties, they do not provide any guidance on how practitioners should relax the equivariant constraints in a way that improves optimization.\n- The analysis in this paper utilizes gradient flow for optimization. In practice, most models are trained with much noisier optimization regimes, e.g., stochastic gradient descent. This creates a gap between the theoretical results shown in this paper and what a practitioner expects to see in practice.", "was_revised": true, "final_justification": "I believe that this work provides a novel perspective on possible optimization difficulties in equivariant networks that would be interesting to the community. While the original submission had some issues in the formulation of the theorems, as also pointed out by the area chair, I believe that the authors did a good job resolving them during the rebuttal period. Due to that, and given that the authors will do the appropriate corrections discussed during the discussion with the reviewers and area chair, I keep my rating of accept.", "rebuttal": "We thank the reviewer for their positive evaluation of our work! We appreciate that the reviewer finds our work timely, the presentation clear, and the experimental setup supportive and insightful. We hope our response addresses the weaknesses and questions raised.\n\n## Weaknesses:\n* Thanks for this feedback! Indeed the current work focuses on identifying an interesting mechanism (hidden symmetries) which generates critical points and even spurious minima in constrained models (e.g. equivariance). While providing explicit guidance on relaxation techniques is beyond the scope of our work, we can provide suggestions for mitigating the effects of the spurious minima.\n\n     In practice, we expect increased width to overcome the spurious minima issues in constrained networks. For wide networks, we expect there to be enough transitive blocks initialized on the “good” side so a network could learn to only use the “good” blocks and ignore the “bad” blocks. Further, it turns out if a previous layer is wide, the dimension condition of Theorem 3.3 is less likely to hold.\n\n* We would like to clarify that the theoretical portion focuses on the loss landscape which is independent of optimization methods. Figure 2 depicts a gradient argument to give intuition for Theorem 3.3 but the proof does not rely on gradients. Assuming a nice parameterization, we do not expect SGD to cross the hidden fixed point space. This is because the loss landscape generated from a subset of samples is still subject to the same parameter symmetries which generated the fixed point space. Whether SGD can help for other parameterizations (equivalently changing learning rates on individual parameters) is an interesting question. However, we still expect that a network gets stuck when initialized too close to a bad minima.\n\n     In addition, we ran some experiments using Adam instead of gradient descent and obtained similar results to Figure 5 already in the paper.\n\n## Questions\n* See weaknesses 1\n* See weaknesses 1. We also want to add that the hidden symmetry mechanism described for permuting neurons applies to every layer so we expect increasing depth to have minimal impact on those results.\n* See weaknesses 2\n\n     Finally, we would like to add that Lemma 3.2 and Theorem 3.3 lays the foundation for future work which may investigate how the number of non-global minimum critical points grows as we overparameterize, similar in spirit to [2]. This could potentially give insight on differences in scaling behavior between equivariant and nonequivariant models.\n\nWe thank you again for your time in reviewing our work. Please let us know if you have any other concerns.\n## References\n\n[1] Dauphin, Yann N., Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. \"Identifying and attacking the saddle point problem in high-dimensional non-convex optimization.\" Advances in neural information processing systems 27 (2014).\n\n[2] Simsek, Berfin, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, and Johanni Brea. \"Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances.\" In International Conference on Machine Learning, pp. 9722-9732. PMLR, 2021."}, {"review_id": "O39iKXkW4g", "reviewer": "Reviewer_jLby", "summary": "The authors identify that global parameter symmetries in networks without symmetry constraints may become “hidden symmetries” when restricted to equivariant subspaces. This can cause gradient descent to get trapped in a spurious local minima. The phenomena is studied for shallow networks using permutation representations in a teacher-student setting, where they show that relaxing the constraint changes the spurious minima to a saddle point.", "questions": "- Does the identified phenomena occur for real equivariant networks? Beyond a teacher-student setting?\n- Are the other loss landscape problems where your tools might be useful?\n- Does weight decay and/or optimizers that normalise gradients solve this issue?", "limitations": "none", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n- Very clearly explain a novel phenomenon where equivariant constraints for a model “interact” with parameter symmetries to induce spurious minima.\n- Introduce a new tool to understand the loss landscape of constrained models, where viewing the model as a member of a larger class of unconstrained models can identify “hidden symmetries” that can e.g. cause spurious minima.\n- The student-teacher experiments are well constructed and support the theoretical arguments.\n\n### Weaknesses\n\n- The focus is on transitive permutation representations, and it’s not clear at all how well the tools / problems will generalize to practical models (e.g., this issue is mentioned in the conclusion w.r.t. tensor products). Likewise, the Hessians of “real” GNNs are typically degenerate, so it’s not clear how relevant hidden symmetry induced minima are in “practice.” Similar complains can be made for the co-dimension assumption.", "was_revised": true, "final_justification": "- The paper has significant conceptual novelty.\n- I disagree with the AC that because the experiments lack certain assumptions used in the theoretical results, they are invalidated. I agree with the authors and reviewer AZWo that showing that the conceptual contribution extends to a \"messier\" \"real-world\" setting in fact makes this a potentially stronger contribution.\n\n- I would like to strongly encourage the authors to follow the ACs recommendations, and communicate Thm 3.3 with the proper limitations (from my reading, I also assume it can be fixed).", "rebuttal": "We thank the reviewer for their positive evaluation of our work! We appreciate that the reviewer finds our explanation of the novel hidden symmetry phenomenon clear and the experiments well constructed. We hope our response addresses the weaknesses and questions raised.\n\n## Weaknesses\n* First, while we do focus on models with permutation representations, we emphasize that the results in Section 3.1 are independent of equivariance and these results apply to any architecture we can view as a constrained version of a more general model.\n\n     Whether hidden symmetry induced minima happen in practice is a very good question. First, we would appreciate a reference for degenerate Hessians in real GNNs, is this an empirical observation or provable one? In addition, the proof of Theorem 3.3 does not actually require all Hessians to be degenerate, just that the one used in the hidden fixed point space is nondegenerate. The current assumption that all critical points are degenerate was made to simplify the statement of the theorem but can be changed in the revised version. The assumption on dimensions is the most restrictive one.\n\n     As for practical usefulness, we do also have some results on $S^2$ activations (Appendix C.3), a component used in real equivariant models such as EquiformerV2. There are conditions where spurious minima arise and we sometimes empirically still see such minima even when the dimension condition of Theorem 3.3 is not satisfied.\n\n     Finally, while complaints about applicability of Theorem 3.3 are more justified due to the dimension assumption, we emphasize that Lemma 3.2 is very general and has very mild assumptions. We believe the identification of these hidden symmetry induced critical points is very important even if they are not minima or imply minima (as happens for the additional criteria of Theorem 3.3). This is because saddle points are a known problem for optimization [1].\n\n## Questions\n* See weaknesses\n\n* This is a great question! Again, we emphasize that the hidden symmetry idea of Section 3.1 is independent of equivariance and could be applied to other types of networks.\n\n     Further, we would like to add that Lemma 3.2 and Theorem 3.3 lays the foundation for future work which may investigate how the number of non-global minimum critical points grows as we overparameterize, similar in spirit to [2]. This could potentially give insight on differences in scaling behavior between constrained and unconstrained models.\n\n* This is a very interesting question! Before answering, we want to emphasize that our work focuses on describing the loss landscape by characterizing critical points and minima rather than navigating optimization on the landscape (eg. gradient descent). The latter has a number of subtleties including a crucial dependence on parameterization [3].\n\n     As for the usefulness of weight decay and gradient normalization, we expect these will not help. If we model weight decay with L2 regularization, we note that the parameter symmetries will still be symmetries of the loss function and so it should not affect our analysis. For gradient normalization, we expect no component perpendicular to the boundary between good and bad regions in the first place and normalizing should not affect this. We also tried using Adam instead of gradient descent and found qualitatively similar results to those in Figure 5 and can add these results in the appendix.\n\nWe thank you again for your time in reviewing our work. Please let us know if you have any other concerns.\n\n## References\n\n[1] Dauphin, Yann N., Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. \"Identifying and attacking the saddle point problem in high-dimensional non-convex optimization.\" Advances in neural information processing systems 27 (2014).\n\n[2] Simsek, Berfin, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, and Johanni Brea. \"Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances.\" In International Conference on Machine Learning, pp. 9722-9732. PMLR, 2021.\n\n[3] Kristiadi, A., Dangel, F., & Hennig, P. (2023). The geometry of neural nets' parameter spaces under reparametrization. Advances in Neural Information Processing Systems, 36, 17669-17688."}, {"review_id": "ozCVqofvct", "reviewer": "Reviewer_Xb7d", "summary": "This work studies the effect of equivariant constraints of the loss landscape of neural networks. Their main result establishes that hidden fixed point space can split the loss landscape of neural networks in two. This creates two unconnected regions, hindering efficient optimization if the global optimum is in the opposite region of initialization. They apply this result to permutation equivariant networks (subset of MLPs) by characterizing their hidden fixed point space. Finally they empirically confront their result to the training of small MLPs by highlighting the existence of such spurious minima and that removing the constraint can improve the optimization.", "questions": "- Some notations were undefined. For-example I could not see where $V_n$ (line 95) was defined. In the same way $\\Theta$ (line 128) was not defined: is it a vector space, a group, a ring? I assume it is a vector space of the trainable parameters but it should be defined. Finally, in theorem 3.3, what is the definition of dimension $dim$? Is it Haussdorf, Minkowsky dimension or the usual dimension of a subvector space? This would assume that $\\Theta_H$ is a sub-vector space. Is it proved if used? Finally where do you use the asssumption on the dim in the proof? (page 17)\n- Can you provide some discussion on the satisfiability of the various assumptions used in your theoretical results? For example theorem 3.3 assumes \"Suppose dim(  Θ_H ) = dim( Θ) − 1\" that \"there exists some C such that for any direction r ∈  Θ we have r · ∇L(C r) > 0\" and that \"Hessians of all critical points are nondegenerate.\" These are three assumptions that could be potentially very strong. It would be nice to add some discussion on their satisfiability in some practical or toy cases.\n- One more open question: what is the effect of over-parametrization on these spurious local minima? Can over-parametrization erase these or make these be asymptotically global minima in the limit of large width?\n- Finally I suggest adding a related work section since related work was only mainly discussed in the introduction.", "limitations": "The authors briefly discussed limitations in the introduction. A main limitation in my opinion is that this work focuses on permutation representation, while there is a significant body of works focusing on rotation or translation symmetries. Could the author add some discussion on how would it impact the theory? Are there major limitations in adapting the proofs?", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- the motivation for this work is clearly explained in sections 1, 2.\n- the focus of this work on whether equivariance constraints can create spurious minima seems new and theoretically interesting.\n- the paper is overall well-written, with self-contained mathematical lemmas and propositions.\n- the main results (thm 3.3, 3.4, 3.5) are theoretically interesting and confronted to experiments in section 4 especially figures 5, 6, 7.\n\n**Weaknesses**\n\n- some notations and mathematical objects are undefined which make it hard to fully understand the assumptions used.\n- Assumptions are correctly stated but there is no discussion on their practical aspect.", "was_revised": true, "final_justification": "I believe the results of this work are interesting and novel. \n\nThe main weaknesses to me are:\n\n*  the strength of the assumptions used in the statement of theorem 3.3.\n*  the assumption lacking to theorem 3.3 to apply lemma 3.2. On that point, it is my understanding that although it requires an additional potentially quite strong assumption in the proof, it is a fixable issue.\n\nFinally, even though the assumptions of theorem 3.3 clearly do not apply to deep neural networks, I believe that it is not disqualifying for experiments to not satisfy the assumptions of the proof.", "rebuttal": "We thank the reviewer for their insightful comments. We appreciate that the reviewer finds our work well motivated, novel, theoretically interesting, and well written. We hope our response addresses the weaknesses and questions raised.\n\n## Weaknesses:\n* Thank you for letting us know and for pointing out some of this in the questions! We plan to add a section in the appendix explaining our notation. We will also double check and update the main body to make sure we define everything.\n* Thanks for raising this point! We will make sure to address how strong our assumptions are in the revised version.\n\n     In particular, we would like to highlight that Lemma 3.2 is very general and can be applied for any linearly constrained model regardless of width and depth. We believe the identification of these hidden symmetry induced critical points is very important even if they are not minima or imply minima (as happens for the additional criteria of Theorem 3.3). This is because saddle points are a known problem for optimization [1].\n\n## Questions:\n* Here, $V_n$ is supposed to be the vector space on which we define our group representation and you are correct for $\\Theta$. We mean the usual dimension of a vector space by $\\mathrm{dim}$. That $\\Theta_H$ is a subvector space is elementary enough that we believe it can be omitted. However, your comment did make us realize we assume a linear group action which we will add in the revised version. We have written a short proof below.\n\n     > For any $\\theta_1,\\theta_2\\in\\Theta_H$, any $a_1,a_2\\in\\mathbb{R}$, and any $h\\in H$ we have $h(a_1\\theta_1+a_2\\theta_2)=a_1h\\theta_1+a_2h\\theta_2=a_1\\theta_1+a_2\\theta_2$ where we used linearity of the group action. Therefore $a_1\\theta_1+a_2\\theta_2\\in\\Theta_H$ so $\\Theta_H$ is a vector space.\n\n     This is a very good question! The assumption is used in line 616 where we state the boundary consists of $S_1$ and $\\bar{B}_C(0)$.\n\n* This is a very good suggestion! The dimension condition is quite strong. However the other conditions are quite weak. We would also like to highlight that the dimension condition is only necessary for Theorem 3.3. Lemma 3.2 only requires a mild geometric condition which is broadly applicable.\n\n* This is a great question! We expect increased width to mitigate the spurious minima issue. First, in some cases the dimension condition is less likely to be satisfied so the spurious minima is no longer guaranteed. Secondly, with wider layers, there is a greater chance that some components of the model are initialized in the “correct” half so the model can learn to prioritize those components.\n\n     Finally, we would like to add that Lemma 3.2 and Theorem 3.3 lays the foundation for future work which may investigate how the number of non-global minimum critical points grows as we overparameterize, similar in spirit to [2]. This could potentially give insight on differences in scaling behavior between equivariant and nonequivariant models.\n\n* Thanks for this suggestion! We will make sure to do this, if accepted, with the extra content page\n\n## Limitations\n* We want to emphasize that the hidden symmetry mechanism in Section 3.1 is quite general and independent of equivariance. While we do focus on applying this to finite groups and permutation representations in the rest of the main body, we do also have some results for rotation symmetry in Appendix C.3. In that case, we show $S^2$-activations which use spherical signals (an analog of a permutation representation) can also suffer from the same problems identified in the paper. Note that $S^2$ activations are used in real models such as EquiformerV2.\n\n     Finally, the main difficulty for other architectures is figuring out the right way to embed it as a constrained subspace of a more general model class. For permutation representations with pointwise nonlinearities, this is easy since MLPs naturally give the more general class. For translation symmetry, we note that convolutions can also be thought of as a restriction of a MLP and we at least expect Lemma 3.2 to apply. However, finding a useful unconstrained model class for layers with tensor products is more difficult.\n\nWe thank you again for your time in reviewing our work. Please let us know if you have any other concerns.\n\n## References\n\n[1] Dauphin, Yann N., Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. \"Identifying and attacking the saddle point problem in high-dimensional non-convex optimization.\" Advances in neural information processing systems 27 (2014).\n\n[2] Simsek, Berfin, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, and Johanni Brea. \"Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances.\" In International Conference on Machine Learning, pp. 9722-9732. PMLR, 2021."}, {"review_id": "rrEg2uAYtu", "reviewer": "Reviewer_AZWo", "summary": "The authors study the effect of symmetries in the parameter space on the loss landscape of neural networks. The authors argue that any symmetry (that can be what they call a 'hidden symmetry') will induce possibly spurious local minima. More specifically, they prove that if the subspace $\\Theta_H$ of invariant weights under the group divides the parameter space in two half-spaces, any minimum must be accompanied by a distinct one. They showcase that this result goes beyond the relatively clear fact that if $\\theta$ is a minimum and $h$ is a global parameter symmetry, then $h\\theta$ is also one via numerical experiments.", "questions": "1.  If $\\tilde{\\Theta}\\subseteq \\Theta$, mustn't the symmetry group $P$ for $\\Theta$ (as defined in Definition 2.7) be a subgroup of the symmetry group $\\tilde{P}$ of $\\tilde{\\Theta}$ (if applying $p \\in P$ leaves every function $f_\\theta$ unaffected, that is in particular true for $f_\\theta, \\theta\\in \\tilde{\\Theta}$? But if $P \\leq \\tilde{P}$, there are no $H$ with $H\\leq P$ but $H\\not\\leq P$.\n  In the paragraph just before Definition 2.7, it seems like the authors use a different definition for global parameter symmetries for the set $\\tilde{P}$: A parameter symmetry here seems to be defined as the $h$ so that $h\\tilde{P}\\subseteq \\tilde{P}$. Can the authors clarify?\n\n2. Probably cause by the above, I do not understand what the hidden symmetries are in the numerical experiments. In the text, the authors say that these are the 'permutations within the block'. In which ways are these permutations defined? It seems to me that the only ways the weights in the layer depicted in Figure 4 can be permuted within the block while keeping the function the same is by permuting the rows and columns simultaneously. However, for this action, the matrix stays the same for all values of $\\theta_1$ and $\\theta_2$, so that this does not seem to be a 'hidden symmetry'. It is in particular not clear to me why the subspace $\\theta_1=\\theta_2$ is invariant in this case (which seems to be crucial in the proof of Theorem 3.3). Does not figure 5(a) indicate that it is not?\n\n3. The formulations of Theorem 3.4 and 3.5 are different in the appendix and main paper. In the main paper, 'neuron blocks' are permutation representations, where as 'neuron points' are permutation representations. Which one is correct? It is for me slightly unclear in which sense a single point in parameter space (which I assume is what is meant by a neuron point) can be a representation of the permutation group. On the other hand, reducability seems to be defined for neuron points, making the meaning of reducability of neuron blocks in the formulation in the main paper hard to understand. Can the authors clarify?\nAlong the same lines, when reparametrizing $(\\theta_1,\\theta_2) \\mapsto (\\sqrt{n-1}\\theta_1,\\theta_2)$, the invariant subspace should change ($\\theta_1=\\theta_2$ is not invariant under that reparametrization), but according to Figure 5(b), it does not. Can the authors clarify?\n\n4. Can the authors comment between the relation between their work and [1] (see above)?\n\n5. In the appendix, the authors claim that spherical signals are permutation representations, but then seem to argue that $\\mathrm{SO}(3)$ acts on functions on the sphere. There seems to be a mismatch here, can the authors explain?", "limitations": "The authors could make it clearer that the condition $\\mathrm{dim}(\\widetilde{\\Theta}_H)= \\mathrm{dim}(\\widetilde{\\Theta})-1$ is somewhat restrictive. In the cases the authors consider in their figures and experiments, $\\dim(\\Theta)$ is often 2, so that $\\mathrm{dim}(\\widetilde{\\Theta}_H)= \\mathrm{dim}(\\widetilde{\\Theta})-1$ is plausible. It is unclear what the situation is in more involved cases.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The main  very interesting observations. The fact that symmetries seem to induce spurious local minimum is a new and potentially impactful result. The proof of Theorem 3.3 and the lemmata leading up to it are easy to read and correct. The proofs of Theorem 3.4 and 3.5 also seem correct, although there are some questions about their formulation, see questions below.\n\nThe results (and idea of viewing equivariant nets as restrictions of nets in larger parameter spaces) seem to have tight relations with [1], which is about the relation between training a non-constrained neural network on group-symmetric (augmented) data and the 'layerwise restriction strategy' to constructing equivariant networks. While that work does not speak about hidden symmetries (and in particular do not arrive at a result as 3.3), many of the geometrical arguments seem similar. In particular, [1] also prove a version of Lemma 3.2 in their setting.\n\nWhere the article lacks is in its readability. The authors start by motivating their work by the debate about whether equivariance should be obtained via restriction of architectures or by relying on symmetries in the data, but then quickly transition to mainly focus on other types of symmetry (namely those obtained by transforming the weights of the network). This is not wrong, but is slightly confusing.\nAlong the same lines, there are some crucial concepts that could be explained more thoroughly. In particular, the definition of hidden symmetries is for me slightly unclear. See more detailed questions below.\n\nPossibly related to the above, there are some aspects of the numerical experiments that are highly unclear to me, making me doubt their relevance to the theory part of the paper. I have included some questions about this below also.\n\nThe above outlined unclarities are significant enough for me to not be able to recommend acceptance, but I am very willing to change my score if the question marks are straightened out during rebuttal.\n\n[1] Nordenfors, O., Ohlsson, F., & Flinth, A. (2025). Optimization dynamics of equivariant and augmented neural networks. Transactions on Machine Learning Research.", "was_revised": true, "final_justification": "The author's rebuttal made the meaning of the paper much clearer to me, and I can know appreciate it a lot better. With this added understanding, I now definitely believe that it should be accepted.", "rebuttal": "We thank the reviewer for their detailed and insightful comments. We appreciate that the reviewer finds our results novel and potentially impactful. We address the questions as they detail the concerns raised as weaknesses. We hope this clears some of the confusion.\n\n## Questions\n1. This is a very good question and is the heart of how we define hidden symmetries. You are absolutely correct that for any $\\tilde{\\theta}\\in\\tilde{\\Theta}$ and $p\\in P$ we have $f_{p\\tilde{\\theta}}=f_{\\tilde{\\theta}}$. The part that can break is line 128 in Definition 2.7, we require $p$ to be a homeomorphism. While originally $p:\\Theta\\to\\Theta$, when restricting to $\\tilde{\\Theta}$ all we know is $p\\vert_{\\tilde{\\Theta}}:\\tilde{\\Theta}\\to p(\\tilde{\\Theta})$. The range of $p(\\tilde{\\Theta})$ is not necessarily $\\tilde{\\Theta}$! So $p|_{\\tilde{\\Theta}}$ is not necessarily a homeomorphism! We mention that this is the issue in line 180-181. A concrete example is a permutation transforming the weights in Figure 6a and 6c. Both weights describe the same function but the weight sharing structure is different.\n\n      We do not quite understand the question about the paragraph before Definition 2.7. We only start using $\\tilde{P}$ in Section 3.1. We would appreciate clarification about what the reviewer meant.\n\n2. Thanks for asking! First we clarify what happens when permuting neurons. Suppose the $W_\\ell$ are the ingoing weights to neurons in layer $\\ell$ and the outgoing weights are $W_{\\ell+1}$. Then if we permute the neurons, we transform $W_\\ell\\to PW_\\ell$ and $W_{\\ell+1}\\to W_{\\ell+1}P^T$, that is we permute the rows of the ingoing weights and columns of the outgoing weights, two **different** matrices. Perhaps the reviewer is confusing this parameter symmetry with equivariance constraints, in which we transform both rows and columns of the same matrix.\n\n     As an example, in Figure 4, $W_\\ell$ are the depicted weights and $W_{\\ell+1}$ is a constant matrix unaffected by any permutations. In particular, permuting the first 3 rows of $W_\\ell$ is a hidden symmetry and would bring us away from the constrained subspace (see difference between Figure 6a and 6c, both would give the exact same function but 6c has a different weight sharing scheme and does not lie in the original equivariant subspace). It is only where $\\theta_1=\\theta_2$ that this permutation has no effect.\n\n3. We apologize for the confusion. Your interpretation of neuron point is correct. We originally borrowed terminology from [2] which uses the term neuron points and defines irreducibility of neuron points. What we intended for a neuron block was a neuron point that satisfies the equivariance constraints corresponding to a specific choice of transitive permutation representation on a set of neurons. Perhaps the following is more clear:\n\n     > Suppose we define a transitive permutation rep on a block of $m$ neurons. Consider a $m$-neuron point which satisfies the corresponding equivariance constraints. If this $m$-neuron point is reducible, then the reduced $n$-neuron point satisfies the equivariance constraints for a corresponding transitive permutation rep on the reduced block of $n$ neurons. Further $n|m$.\n\n     Yes, you are correct that the subspace is no longer $\\theta_1=\\theta_2$ but $\\sqrt{n-1}\\theta_1=\\theta_2$. In Figure 5b, this is why the boundary line does not have slope $1$ but instead slope $\\sqrt{2}$.\n\n4. Thanks for bringing up this work! In [1], the main results are how critical points of unconstrained, equivariant, and augmented networks are related, followed by a discussion of their respective stabilities. However, there is no discussion on how these critical points arise which is why this result is different from Lemma 3.2.\n\n     In our work, the focus is on describing a general mechanism (hidden symmetries and hidden fixed point spaces) which gives rise to critical points (Lemma 3.2) and can even imply the existence of additional minima (Theorem 3.3). We view the hidden symmetry mechanism in Lemma 3.2 as a generalization of obtaining critical points from regular parameter symmetries [2,3]. The hidden symmetry mechanism is then applied to equivariant MLP layers built with permutation representations (Theorems 3.4 and 3.5 + experiments).\n\n     One additional note we want to make is that [1] focuses on a particular equivariant subspace since it assumes a specific choice of representation on the hidden layers. In our work, we have the interesting empirical observation that unconstrained models can learn to change the representation in the hidden layers.\n5. This is a great question! What we mean is that spherical signals satisfy a generalized version of permutation representation.\n\n     In a more general definition, we could consider an abstract permutation representation as homomorphism $G\\to\\mathrm{Aut}(X)$ where $X$ is some object (maybe a set, or maybe it has some additional structure). In the case where $X$ is a finite set, then $\\mathrm{Aut}(X)\\cong\\mathrm{Sym}(|X|)$ and we recover the usual definition of an abstract permutation representation. In the case of spherical signals, we take $X$ to be $S^2$ and the abstract permutation representation is the action of rotations on the sphere.\n\n     To move from an abstract perm rep to a perm rep, we then construct a vector space (over field $\\mathbb{R}$ for us) with elements of $X$ as the basis. Then any point in the space is of the form $\\sum_{x\\in X}c_x x$ where $c_x\\in\\mathbb{R}$ are the coefficients. But we can also interpret $c_x$ as a function $X\\to\\mathbb{R}$. For finite $X$, the induced representation on $\\\\{X\\to\\mathbb{R}\\\\}$ gives us the usual $|X|$-dimensional perm reps. For infinite $X$ such as $S^2$, we instead have an induced representation on the infinite dimensional vector space of $\\\\{S^2\\to\\mathbb{R}\\\\}$ which corresponds precisely to the spherical signals.\n\n     We can make this point more clear in the appendix.\n\n## Limitations\nThanks for this suggestion. Indeed the dimension condition is somewhat restrictive. We plan to add a discussion of how restrictive each of these conditions are.\n\nHowever, we would like to point out that Lemma 3.2 is very general and can be applied for any linearly constrained model regardless of width and depth. We believe the identification of these hidden symmetry induced critical points is very important even if they are not minima or imply minima (as happens for the additional criteria of Theorem 3.3). This is because saddle points are a known problem for optimization [4].\n\nWe thank you again for your time in reviewing our work. Please let us know if you have any other concerns.\n\n## References\n\n[1] Nordenfors, Oskar, Fredrik Ohlsson, and Axel Flinth. \"Optimization Dynamics of Equivariant and Augmented Neural Networks.\" Transactions on Machine Learning Research.\n\n[2] Simsek, Berfin, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, and Johanni Brea. \"Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances.\" In International Conference on Machine Learning, pp. 9722-9732. PMLR, 2021.\n\n[3] Fukumizu, Kenji, and Shun-ichi Amari. \"Local minima and plateaus in hierarchical structures of multilayer perceptrons.\" Neural networks 13, no. 3 (2000): 317-327.\n\n[4] Dauphin, Yann N., Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. \"Identifying and attacking the saddle point problem in high-dimensional non-convex optimization.\" Advances in neural information processing systems 27 (2014)."}]

Author rebuttal:

Meta-review: The paper considers the following setting:

- $\mathcal{F}\_\theta = \\{f\_\theta : X \to Y \mid \theta \in \Theta\\}$ is a class of functions over the parameter space $\Theta$, and $L:\Theta \to \mathbb{R}$ a loss function.

- $H\le P$, where $P$ is a group of global symmetries (i.e., $f_\theta = f_{g\theta}$ for all $g \in P$ and $\theta \in \Theta$).

- $\mathcal{F}_{\tilde{\Theta}}$ is a class of functions corresponding to $\tilde{\Theta} \subset \Theta$, interpreted as the usual restriction to weights that yield equivariant layers.

The main result concerns conditions under which two minima may be found on $\tilde{\Theta}$. 
Specifically, assuming that: the loss function is $C^1$, there exists a minimum 
 $\theta_1 \notin\Theta\_H\cap \tilde{\Theta}$ for $L|\_{\tilde{\Theta}}$, 
$\mathrm{codim}(\Theta_H \cap \tilde{\Theta}) = 1$, there exists a constant $C$ such that 
for any direction $\hat{r} \in \tilde{\Theta}$ one has $\hat{r} \cdot \nabla L(C\hat{r}) > 0$, 
Hessians of critical points are nonsingular, and $(\Theta_H \cap \tilde{\Theta}) \oplus (\Theta_H^\perp \cap \tilde{\Theta}) = \tilde{\Theta},$ 
then there exists a second minimum $\theta_2 \neq \theta_1$ for $L|_{\tilde{\Theta}}$. 
The result is subsequently specialized to cases involving permutation representations. 
The experimental section presents an experiment in the teacher–student setting, illustrating a scenario where the theorem’s implication appears to hold without satisfying the stated assumptions.

Several issues were raised and communicated to the authors relating to core definitions, the formulation, proofs (Lemma 3.2 and Theorem 3.3) and scope of the main result, as well as its empirical validation. In particular, reviewers expressed concerns regarding the practical applicability of the results, citing the codimension and nondegeneracy assumptions as well as the numerical experiments. During the discussion, it was remarked that in the experimental section several assumptions underlying the theoretical result were not verified, and some were violated, leaving the empirical verification of the result open. It was further noted that the minima observed had been reported and studied in prior work. The authors have addressed these issues to the satisfaction of the reviewers, who appreciated the importance of the problem concerning the optimization of equivariant networks, the stated result regarding the existence of an additional minimum in the subspace $\tilde{\Theta}$, and the empirical results on the relaxation of layer equivariance.
Decision: Accept (poster)