Title: A Tale of Two Symmetries: Exploring the Loss Landscape of Equivariant Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: equivariance

loss landscape

optimization

group theory

parameter symmetry

symmetry
Abstract: Equivariant neural networks have proven to be effective for tasks with known underlying symmetries. However, optimizing equivariant networks can be tricky and best training practices are less established than for standard networks. In particular, recent works have found small training benefits from relaxing equivariance constraints. This raises the question: do equivariance constraints introduce fundamental obstacles to optimization? Or do they simply require different hyperparameter tuning? In this work, we investigate this question through a theoretical analysis of the loss landscape geometry. We focus on networks built using permutation representations, which we can view as a subset of unconstrained MLPs. Importantly, we show that the parameter symmetries of the unconstrained model has nontrivial effects on the loss landscape of the equivariant subspace and under certain conditions can provably prevent learning of the global minima. Further, we empirically demonstrate in such cases, relaxing to an unconstrained MLP can sometimes solve the issue. Interestingly, the weights eventually found via relaxation corresponds to a different choice of group representation in the hidden layer. From this, we draw 3 key takeaways. (1) By viewing the unconstrained version of an architecture, we can uncover hidden parameter symmetries which were broken by choice of constraint enforcement (2) Hidden symmetries give important insights on loss landscapes and can induce critical points and even minima (3) Hidden symmetry induced minima can sometimes be escaped by constraint relaxation and we observe the network jumps to a different choice of constraint enforcement. Effective equivariance relaxation may require rethinking the fixed choice of group representation in the hidden layers.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the effect of symmetries in the parameter space on the loss landscape of neural networks. The authors argue that any symmetry (that can be what they call a 'hidden symmetry') will induce possibly spurious local minima. More specifically, they prove that if the subspace $\\Theta_H$ of invariant weights under the group divides the parameter space in two half-spaces, any minimum must be accompanied by a distinct one. They showcase that this result goes beyond the relatively clear fact that if $\\theta$ is a minimum and $h$ is a global parameter symmetry, then $h\\theta$ is also one via numerical experiments.\n\nSTRENGTHS AND WEAKNESSES:\nThe main  very interesting observations. The fact that symmetries seem to induce spurious local minimum is a new and potentially impactful result. The proof of Theorem 3.3 and the lemmata leading up to it are easy to read and correct. The proofs of Theorem 3.4 and 3.5 also seem correct, although there are some questions about their formulation, see questions below.\n\nThe results (and idea of viewing equivariant nets as restrictions of nets in larger parameter spaces) seem to have tight relations with [1], which is about the relation between training a non-constrained neural network on group-symmetric (augmented) data and the 'layerwise restriction strategy' to constructing equivariant networks. While that work does not speak about hidden symmetries (and in particular do not arrive at a result as 3.3), many of the geometrical arguments seem similar. In particular, [1] also prove a version of Lemma 3.2 in their setting.\n\nWhere the article lacks is in its readability. The authors start by motivating their work by the debate about whether equivariance should be obtained via restriction of architectures or by relying on symmetries in the data, but then quickly transition to mainly focus on other types of symmetry (namely those obtained by transforming the weights of the network). This is not wrong, but is slightly confusing.\nAlong the same lines, there are some crucial concepts that could be explained more thoroughly. In particular, the definition of hidden symmetries is for me slightly unclear. See more detailed questions below.\n\nPossibly related to the above, there are some aspects of the numerical experiments that are highly unclear to me, making me doubt their relevance to the theory part of the paper. I have included some questions about this below also.\n\nThe above outlined unclarities are significant enough for me to not be able to recommend acceptance, but I am very willing to change my score if the question marks are straightened out during rebuttal.\n\n[1] Nordenfors, O., Ohlsson, F., & Flinth, A. (2025). Optimization dynamics of equivariant and augmented neural networks. Transactions on Machine Learning Research.\n\nQUESTIONS:\n1.  If $\\tilde{\\Theta}\\subseteq \\Theta$, mustn't the symmetry group $P$ for $\\Theta$ (as defined in Definition 2.7) be a subgroup of the symmetry group $\\tilde{P}$ of $\\tilde{\\Theta}$ (if applying $p \\in P$ leaves every function $f_\\theta$ unaffected, that is in particular true for $f_\\theta, \\theta\\in \\tilde{\\Theta}$? But if $P \\leq \\tilde{P}$, there are no $H$ with $H\\leq P$ but $H\\not\\leq P$.\n  In the paragraph just before Definition 2.7, it seems like the authors use a different definition for global parameter symmetries for the set $\\tilde{P}$: A parameter symmetry here seems to be defined as the $h$ so that $h\\tilde{P}\\subseteq \\tilde{P}$. Can the authors clarify?\n\n2. Probably cause by the above, I do not understand what the hidden symmetries are in the numerical experiments. In the text, the authors say that these are the 'permutations within the block'. In which ways are these permutations defined? It seems to me that the only ways the weights in the layer depicted in Figure 4 can be permuted within the block while keeping the function the same is by permuting the rows and columns simultaneously. However, for this action, the matrix stays the same for all values of $\\theta_1$ and $\\theta_2$, so that this does not seem to be a 'hidden symmetry'. It is in particular not clear to me why the subspace $\\theta_1=\\theta_2$ is invariant in this case (which seems to be crucial in the proof of Theorem 3.3). Does not figure 5(a) indicate that it is not?\n\n3. The formulations of Theorem 3.4 and 3.5 are different in the appendix and main paper. In the main paper, 'neuron blocks' are permutation representations, where as 'neuron points' are permutation representations. Which one is correct? It is for me slightly unclear in which sense a single point in parameter space (which I assume is what is meant by a neuron point) can be a representation of the permutation group. On the other hand, reducability seems to be defined for neuron points, making the meaning of reducability of neuron blocks in the formulation in the main paper hard to understand. Can the authors clarify?\nAlong the same lines, when reparametrizing $(\\theta_1,\\theta_2) \\mapsto (\\sqrt{n-1}\\theta_1,\\theta_2)$, the invariant subspace should change ($\\theta_1=\\theta_2$ is not invariant under that reparametrization), but according to Figure 5(b), it does not. Can the authors clarify?\n\n4. Can the authors comment between the relation between their work and [1] (see above)?\n\n5. In the appendix, the authors claim that spherical signals are permutation representations, but then seem to argue that $\\mathrm{SO}(3)$ acts on functions on the sphere. There seems to be a mismatch here, can the authors explain?\n\nLIMITATIONS:\nThe authors could make it clearer that the condition $\\mathrm{dim}(\\widetilde{\\Theta}_H)= \\mathrm{dim}(\\widetilde{\\Theta})-1$ is somewhat restrictive. In the cases the authors consider in their figures and experiments, $\\dim(\\Theta)$ is often 2, so that $\\mathrm{dim}(\\widetilde{\\Theta}_H)= \\mathrm{dim}(\\widetilde{\\Theta})-1$ is plausible. It is unclear what the situation is in more involved cases.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the effect of equivariant constraints of the loss landscape of neural networks. Their main result establishes that hidden fixed point space can split the loss landscape of neural networks in two. This creates two unconnected regions, hindering efficient optimization if the global optimum is in the opposite region of initialization. They apply this result to permutation equivariant networks (subset of MLPs) by characterizing their hidden fixed point space. Finally they empirically confront their result to the training of small MLPs by highlighting the existence of such spurious minima and that removing the constraint can improve the optimization.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- the motivation for this work is clearly explained in sections 1, 2.\n- the focus of this work on whether equivariance constraints can create spurious minima seems new and theoretically interesting.\n- the paper is overall well-written, with self-contained mathematical lemmas and propositions.\n- the main results (thm 3.3, 3.4, 3.5) are theoretically interesting and confronted to experiments in section 4 especially figures 5, 6, 7.\n\n**Weaknesses**\n\n- some notations and mathematical objects are undefined which make it hard to fully understand the assumptions used.\n- Assumptions are correctly stated but there is no discussion on their practical aspect.\n\nQUESTIONS:\n- Some notations were undefined. For-example I could not see where $V_n$ (line 95) was defined. In the same way $\\Theta$ (line 128) was not defined: is it a vector space, a group, a ring? I assume it is a vector space of the trainable parameters but it should be defined. Finally, in theorem 3.3, what is the definition of dimension $dim$? Is it Haussdorf, Minkowsky dimension or the usual dimension of a subvector space? This would assume that $\\Theta_H$ is a sub-vector space. Is it proved if used? Finally where do you use the asssumption on the dim in the proof? (page 17)\n- Can you provide some discussion on the satisfiability of the various assumptions used in your theoretical results? For example theorem 3.3 assumes \"Suppose dim(  Θ_H ) = dim( Θ) − 1\" that \"there exists some C such that for any direction r ∈  Θ we have r · ∇L(C r) > 0\" and that \"Hessians of all critical points are nondegenerate.\" These are three assumptions that could be potentially very strong. It would be nice to add some discussion on their satisfiability in some practical or toy cases.\n- One more open question: what is the effect of over-parametrization on these spurious local minima? Can over-parametrization erase these or make these be asymptotically global minima in the limit of large width?\n- Finally I suggest adding a related work section since related work was only mainly discussed in the introduction.\n\nLIMITATIONS:\nThe authors briefly discussed limitations in the introduction. A main limitation in my opinion is that this work focuses on permutation representation, while there is a significant body of works focusing on rotation or translation symmetries. Could the author add some discussion on how would it impact the theory? Are there major limitations in adapting the proofs?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

comments: SUMMARY:
The authors identify that global parameter symmetries in networks without symmetry constraints may become “hidden symmetries” when restricted to equivariant subspaces. This can cause gradient descent to get trapped in a spurious local minima. The phenomena is studied for shallow networks using permutation representations in a teacher-student setting, where they show that relaxing the constraint changes the spurious minima to a saddle point.

STRENGTHS AND WEAKNESSES:
### Strengths

- Very clearly explain a novel phenomenon where equivariant constraints for a model “interact” with parameter symmetries to induce spurious minima.
- Introduce a new tool to understand the loss landscape of constrained models, where viewing the model as a member of a larger class of unconstrained models can identify “hidden symmetries” that can e.g. cause spurious minima.
- The student-teacher experiments are well constructed and support the theoretical arguments.

### Weaknesses

- The focus is on transitive permutation representations, and it’s not clear at all how well the tools / problems will generalize to practical models (e.g., this issue is mentioned in the conclusion w.r.t. tensor products). Likewise, the Hessians of “real” GNNs are typically degenerate, so it’s not clear how relevant hidden symmetry induced minima are in “practice.” Similar complains can be made for the co-dimension assumption.

QUESTIONS:
- Does the identified phenomena occur for real equivariant networks? Beyond a teacher-student setting?
- Are the other loss landscape problems where your tools might be useful?
- Does weight decay and/or optimizers that normalise gradients solve this issue?

LIMITATIONS:
none
TITLE: Official Review
CLARITY: 4
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work investigates the question of whether equivariant constraints in neural networks can create optimization challenges. First, the authors provide an extensive presentation of the preliminary background that is necessary for their analysis. Then, they focus on the analysis of equivariant models that can be derived by constraining the parameters of a fully unconstrained model to a specific parameter subspace. They define the concept of hidden parameter symmetries, which are symmetries in the unconstrained space that disappear in the equivariant subspace, and they show how these can induce critical points or spurious minima in the loss landscape of the constrained equivariant model. Additionally, the authors construct a simple example of a single-layer model where these critical points can be observed experimentally, and they demonstrate how the parameter space can be split into two distinct halves that cannot be traversed by gradient flow. As a result of the theoretical analysis and simple experimental observations, the authors conclude that, indeed, partially removing the equivariant constraints can result in an easier optimization landscape, allowing the network to utilize different types of latent equivariant representations and traverse distinct equivariant parameter subspaces.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The topic of this work is timely since there are multiple works that showcase evidence supporting the benefits of symmetry breaking in equivariant neural networks. Additionally, while most previous works provide mostly experimental evidence, this paper presents a theoretical hypothesis on the possible mechanism behind this phenomenon.\n- The presentation of the theoretical analysis is clear and can be easily understood even by readers who might not be experts in the topic.\n- The simple experimental setup provides both evidence supporting the theoretical hypothesis and also helps the readers build a better intuition on the problem at hand.\n\nWeaknesses:\n- While the authors provide a possible mechanism showcasing how an equivariant constraint can create optimization difficulties, they do not provide any guidance on how practitioners should relax the equivariant constraints in a way that improves optimization.\n- The analysis in this paper utilizes gradient flow for optimization. In practice, most models are trained with much noisier optimization regimes, e.g., stochastic gradient descent. This creates a gap between the theoretical results shown in this paper and what a practitioner expects to see in practice.\n\nQUESTIONS:\n- Can the results of this work be used to provide a guideline on how to relax the equivariant constraint of a given model?\n- How reasonable is it to expect that the separation of the parameter space into two halves can also happen in deeper models with a much higher number of parameters?\n- How do these critical points induced by hidden symmetries affect optimization if, instead of gradient flow, stochastic gradient descent is used?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the consequences for considering the loss-landscape of designing neural networks to be equivariant. This is done with an MLP as a case study. The authors adopt the view that equivariant networks should be seen as part of a larger family of un constrained networks. In this case it turns out that the loss landscape of the equivariant networks may have minima which are saddle points in the case of unconstrained networks; a feature which would lead to undesirable training behavior.\n\nIn general the idea is interesting and well founded. I thus recommend that the paper is accepted.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n- The idea is interesting and compelling. I applaud that the authors for writing a paper aimed at principled understanding of a topic.\n- The manuscript is in general well written.\n- The authors comprehensively reflect on the limitations of their method.\n\nWeaknesses:\n- The paper has a variety of central results (Theorem 3.3, 3.4 and 3.5). Unfortunately the proofs of all these important results are moved to the appendix. I would suggest to the authors that they provide either the full proof or a sketch of proof in the main body of the paper.\n- I am slightly confused about section 3.2 overall. As I understand it, it is supposed to characterize hidden fixed point spaces of equivariant networks. However, I do not see the direct connection to the previous sections.\n- I urge the authors to work on the clarity in communication of the ideas proposed. For instance a main point of the paper, as I understand it, is that equivariant networks might introduce minima, which would not appear in a corresponding unconstrained model. I suggest that the authors to attempt to make it even more clear how this happens and why it is a problem, e.g. through further use of illustrations or examples.\n\nQUESTIONS:\n- Line 24-25: I assume it is an error that this sentence is included?\n- Line 85: I suggest that the authors provide a reference to how maximally expressive equivariant linear layers are constructed forn an irrep decomposition.\n- Line 94: Is it assumed implicitly that G is finite?\n- Line 105: Isn't it meant to be $\\rho \\circ \\pi: G \\to GL(V_n)$ instead of $\\rho: G \\to GL(V_n)$ to be consistent with previous notation?\n- Line 107. I assume that the action of $G$ on the set $\\{b_, ..., b_n\\}$ is transitive if it only have one orbit?\n- Line 109: Please provide a reference or a proof for this Lemma.\n- Line 128: Why is it needed that $p$ is a homeomorphism (wrt. the Euclidean topology i assume)? Is it not sufficient to require it to be bijective for P to be a group?\n- Line 152: There seems to be something wrong with the sentence \"Then if $\\Sigma is ...$\".\n- Line 179: By linearly constrained subspace do you mean a fixed point subspace?\n- Line 180: Which group is $p$ an element of and what is the action?\n- As mentioned, a main point of the paper, as I understand it, is that equivariant networks might introduce minima, which would not appear in a corresponding unconstrained model. However do you have any reflections on whether this poses a problem empirically? I ask since often networks will be  extremely overparametrized, and we rarely seek a global minima anyway. As such one could think that the additional minima could be introduced through the use of an equivariant model would not pose a problem.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-07"}