Symmetry in Neural Network Parameter Spaces
Authors: Bo Zhao
Robin Walters
Rose Yu
Venue: Accepted by TMLR ()

Abstract: Modern deep learning models are highly overparameterized, resulting in large sets of parameter configurations that yield the same outputs. A significant portion of this redundancy is explained by symmetries in the parameter space—transformations that leave the network function unchanged. These symmetries shape the loss landscape and constrain learning dynamics, offering a new lens for understanding optimization, generalization, and model complexity  that complements existing  theory of deep learning. This survey provides an overview of parameter space symmetry. We summarize existing literature, uncover connections between symmetry and learning theory, and identify gaps and opportunities in this emerging field.

Published in Transactions on Machine Learning Research (12/2025)
Symmetry in Neural Network Parameter Spaces
Bo Zhao
bozhao Bucsd. eduu
University of California, San Diego
Robin Walters
walters @northeasterne edu
Northeastern University
Rose Yu
Aeseyuuucsd.cm edu
University of California, San Diego
Reviewed on OpenReview: https: openreview. net/ forum? id= 3LpWq50Y61
Abstract
Modern deep learning models are highly overparameterized, resulting in large of param- eter configurations that yield the same outputs. A significant portion of this redundancy is explained by symmetries in the parameter space space- transformations that leave the network function unchanged. These symmetries shape the loss landscape and constrain learning dynamics, offering a new lens for understanding optimization, generalization, and model complexity that complements existing theory of deep learning. This survey provides an overview of parameter space symmetry. We summarize existing literature, uncover connec- tions between symmetry and learning theory, and identify gaps and opportunities this emerging field.
1 Introduction
Despite their remarkable empirical success, neural networks remain not fully understood from thcoretical standpoint. Designing effective architectures often relies heuristics, and training outcomes be highly sensitive to initialization and optimization details. The rapid growth of model sizes eeeess-ieellieee by the re- cent explosion in the size of language and other generative models (Kaplan et al. 2020) further complicates
efforts to characterize training dynamics and generalization behavior. While thcoretical work has clarified aspects overparameterriat ion (Belkin al. 2019 Neyshabur al.) 2019) and implicit regularization gradient descent (Soudry al. 2018), we still lack unified understanding of how the structure of parameter space shapos the loss landscape and affects optimization.
This survey proposes parameter space symmetry as valuable, underexplored perspective for understanding neural networks. Parameter space symmetries are transformations of neural net- work parameters that leave the output unchanged, e.g. per- muting neurons in hidden layer (Figure D. One of the most important consequences of these symmetries is that they in- duces nontrivial structure in the level sets of the loss function. In overparameterized networks, where many parameter config- urations can yield the same function output, the loss landscape oflen contains high dimensional manifolds of global minima CCoppe 2018). Symmetry accounts for much of this degener- acy by mapping parameters within level without changing the network's function. Understanding these equivalence classes is essential for analyzing generalization, op- timization dynamics, and the loss landscape in deep learning.
permute
Figure 1: Example of parameter space sym- metry from neuron permutation. Swap- ping hidden units and their weights yields a different parameter configuration with the same function.

---

Published in Transactions on Machine Learning Research (12/2025)
While symmetry in the data space has been central to geometric deep learning and equivariant models (Bronstein et al. 2021), symmetry in the parameter space has only recently begun to receive sustained attention. Recent work explores parameter symmetry in diverse contexts, including loss landscapes and mode connectivity [Garipoy et al. 2018 Draxler et al., 2018), conserved quantitics and training dynamics (Kunin et al.) 2021: Tanaka & Kunin p. symmetry- based optimization and model averaging (Entezari
ct al. 2022 Ainsworth et al. 2023), and symmetry-aware sampling in Bayesian inference (Wiese et al. (0023). This survey aims to define symmetries in neural network parameter spaces, unify existing theoretical and algorithmic perspectives, and highlight their relevance to learning theory, optimization, and model analysis. We hope that clearer understanding of parameter space symmetries will lead to more principled approaches in deep learning theory and practice.
The rest of this paper is organized as follows. Section reviews definitions and examples of symmetries in the parameter space of neural networks. Section 3 collectively survey the theoretical and algorithmic applications of parameter space symmetry in deep learning: Soction discusses the role of symmetry in the geometry and topology of loss level sets, with focus on minima; Section 4 lists progress in optimization by incorporating the knowledge of parameter space symmetry; Section discusses conserved quantities in gradient flow associated with parameter space symmetry and their applications in understanding lcarning dynamics. Section 0 then discusses how parameter space symmetry interacts with data and representation symmetries, as well as applications in tasks where neural network parameters are treated as data. We conclude with a list of open questions and opportunities in Section
2 Parameter Space Symmetry
In this section, we explore various definitions and examples of parameter space symmetry, starting with transformations of the parameters that preserve the feedforward function. Then we expand our discussion to inchude relaxed dofinitions of symmetry, which preserve the overall loss function or the foodforward function's value on subsets of data. Along with these definitions, we provide a list of known symmetries and show how they arise from the architecture of neural networks. Finally, we discuss the impact of symmetries on parameter identifiability and whether the known symmetries complete.
2.1 Loss-Invariant Parameter Transformations and Symmetry
Let θ be the space of parameters and D be the space of data. In supervised learning, D is decomposed into Dinput Diarget; where Dmpput and Diarget correspond to the space of inputs and target labels, respectively. Let f: ④ x Dimput + Dranget be a neural network function, and c: Dearget X Dtarget + IR be function that measures the discrepancy between the neural network's prediction and the ground truth label. The loss
function L: θ x D - I is the composition of I and c. That is, for (0, (x, E θ x D, we define the loss L(0, (z, == (f( v). With occasional exceptions (Bassey et al. 2021 Huang al., 2018). the
parameter space of neural network is typically real vector space.
Symmetries are transformations that preserve certain properties of an object. In this paper, we focus on parameter space symmetry, which are transformations of a neural network's parameters that preserve loss. Mathematically, this is the set of bijective transformations T: θ + O such that L(w) = L(T(w)). One may consider restricted sets of symmetry by imposing additional constraints such as linearity or smoothness.
The sot of all such transformations forms group under function composition. To see this, we first examine its structure and properties. Viewing the transformations as functions, we are able to define the composition of two transformations. The composition of two bas--invrriant transformations results in another loss- invariitr transfrratiin. Since function compositions are associative, the composition of these transformations are associative. Additionally, the identity transformation is always loss preserving, therefore included in our set. Finally, since each transformation is bijective, an inverse exists for every element in the set. These properties of the set symmetry transformations under composition satisfy the definition of a group.
Definition 2.1 (Group). A group set G together with a composition law that satisfies (1) Associativity: (g) 92) o 33 = gi (g2 ga) for all gi, 92, 93 in G; (2) Identily: There erists an identily element. in G such
2

---

Published in Transactions on Machine Learning Research (12/2025)
(a)
(b)
(c)
(d)
To
TO
Sg 9 1
hxh
GX6
TXT
ri IT =
Ti
T2
50 $1 $2
G
h T
12
12 (Sp r;) (1) on h(g) or h(g') h(g % g¹)
Figure An example of symmetry and group actions. (a) The dihedral group D3, which is the group of symmetries of regular triangle. D3 has six elements: three rotations ro, T2 and three reflections so, $1 $2. (b) The identily element ro does not change the object. Each group element has an inverse. (c) Composition of transformations are associative. (d) A group action can be defined as homomorphism h from a group G to the group of symmetry transformations T.
that go e = 9 and e g- = for all in G; (3) Inverse: For every element in G, there erists an inverse element g such that g gog- and g =c
Example 2.1. We illustrate group of symmetry transformations on the dihedral group D3, which represents symmetrics of regulor triangles (Figure D3 has siz elements: three rotations 09, 120", 240 (ro, (2, and three reflections across azes passing through each verter and the midpoint of the opposite side (so, $1, 82). The composition of these transformations is also symmetry. The identity element does not change the triangle. Each transformation has an inverse, which is also in the group. Composition of transformations are associative.
Following Definition 2.1 the set of all loss-invariant and bijective transformations, which we denote by Ge, forms group. In practice, we typically consider specific subgroups of Ge.L, since the full group often difficult to characterize.
For generality and to simplify analysis, describe symmetry using abstract groups, which focus on the algebraic properties of groups detached from specific transformations, An example that will appear frequently is the n x n general lincar group over R. This group, denoted by GL,, consists of invertible n x n real matrices, with composition defined by matrix multiplication. We will also encounter several subgroups of (R), including the orthogonal group (R) which consists of real matrices whose transpose equal inverse, and the positive scaling group R&,, which consists of diagonal matrices with positive diagonal entries. Another group relevant to neural network parameter spaces is the symmetric group Sn, which consists of permutations of the set
Group actions connect the abstract concept of groups with concrete sets of transformations. A group action is a structure- potuuurrrppeeeviigggg map from a group into a group of transformations (Figure 2
Definition 2.2 (Group action). An aclion of a group G on a set S is a map :: G S that salisfies s-s=s for all S and g (g' () s) = (gg') -s for all g.g' in G and all in S.
A parameter space symmetry then be described the loss unchanged. In many machine learning settings, these group actions are linear. This leads to the concept of a representation, which maps group elements to invertible matrices and enables the group to act on a vector space by linear transformations.
group action the space of parameters that leaves
Definition 2.3 (Representation). representation of a group G is a homomorphism p: - (R., meaning that p(g192) p(g)))((z) for all 91.92 G.
2.2 Functional Neural Network Symmetry
Parameter space symmotry can be defined in various ways, depending on the transformation preserved, i.c. the neural network function or the loss function, and the scope of the data considered, which can range from all possible data, subsets of data, or particular distribution (Figure
3

---

Published in Transactions on Machine Learning Research (12/2025)
This section focuses on the strictest form of parameter space symmetry, which involves transformations that leave the neural network output invariant across all data. Such symmetry preserves the leedforward function. We discuss various general definitions in Section 2.3
Definition 2.4 (Functional neural network symmetry). Let Q be the parameter space of a neural network f: θ D snpuut + Piagget- A parameter space symmetry of f is a (possibly nonlinear) action of a group G on θ that leaves f invariant,
- [ -a--02))-000 [(0,z), Vg EE Vr E D inpput.
The group G is called symmetry group of f-
2.2.1 Examples:
Symmetries in Common Neural Network Components
Different neural network model architectures give rise to different parameter space symmetries. In the following examples, we examine common components of neural networks and identify specific symmetries associated with each. We begin with a linear network, which offers clean setting to illustrate how parameter symmetries emerge from rescaling adjacent layers. We work lowards realistic examples afterwards.
Example 2.2 (Linear). Consider two-layer linear neural network Linear(W,, W1, b2, b) = W2 + bn) + b2, with (W2, W1, b2, bn) E θ = Rmxx x AR x Rhn x Rh and X E B This architecture has a (IR symmetry, acting on θ by g (W2, W1, b2, b1) = (W2g -1, ,, gbi),, for g E (R) since
ftincar (W2, W1, b2 b1), X) = W29 1 (gWW X + gbz) + b2 = W2W X + b,) + by
= fisnear (W2, W1, b2, b1, X).
Symmetries of similar forms appear in networks with activation functions, which are more commonly used than linear networks. In particular, many common activation functions are equivariant under a nontrivial group, which leads to following symmetrics (Figure BD).
Proposition 2.5 (Zhao et al. (2023)). Let o: R² + R² be a function that satisfies (gg)) = p(g))(Z) for a group G and a representation pt G GL (E). Consider a function f: @x - Rxx (W2. W1, b2, b) +
W20 X + b1) b2, where (W2, W1, b2 b1) E θ m RRxx x Rm x R and X B Then, f admits functional parameter space symmetry defined by g- (W2, W1, b1) (Waplg ,2,
The symmetrics in the next three examples result from equivariance of various pointwise activation functions. A function a: A > Re is called pointwise if it is defined as (z)); = (z));) for some scalar functions (o;: R + / In practice, o, is usually the same across all indices i. A pointwise function o: R² + A is homogencous if there exists degree a E TRS such that a(cz), = (()) for all E Ry0 and = E R*. Common homogeneous functions include ReLU, LeakyReLU, and monomials. Since the bias terms are transformed similarly the other weights, we omit the bias terms in the following examples for brevity.
Example 2.3 (Homogeneous Activation). Consider two-layer neural network f(2, W1, W2 with homogencous function a of degree a, where (W2, W1) E θ = mmmxh x hRh and X E B A symmetry group of this architecture is the positive scaling group R/g0r which consists of diagonal matrir with positive diagonal entries and act on θ by g (W2, W1) = (W2g (Wig-".gWin ,gW.), for g E R²S0 (Badrinarayanan et al.) 2015). Example 2.4 (Tanh). Consider a two-layer hyperbolic langent nearul network Stant W X)
W2 tanh(W,X) where (W2, Wi) E θ gmxh x hg and X E Baxk This architecture has a sign- flip symmetrry, ER, that consists of diagonal matrix with all diagonal entries in {1,-1}, acting on θ by 9 (W2, W1) = (W2g"1, (W) [Chen et al. 1993).
Example 2.5 (Radial neural network (Ganev et al. 2022 ). Oflen appearing in equivariant newral networks
[Weiler et al.) 2018: Weiler & Cesa 2019), a radial rescaling activation a: R² + R2 has the form o(z) = [[[x]]) for some function f: R - R. A two-layer neural network f(W,, W1, X)= = W2o with a radial rescaling activation a has an On (RR symmetrr, acting on θ by g (W2, Wi) = (W29 -1 ".9. W)), for g E (R)..
When o is G-invariant, which means a(gZ) oZ), there exists a group action that acts on only the input weights of a (Figure 3). We illustrate this in the next. two examples.
4

---

Published in Transactions on Machine Learning Research (12/2025)
(a)
(b)
(c)
gW,
W2
gW1
Wap(o
=(gW))==(())
=
agW,)) P())((w))))
Figure 3: Parameter space symmetries arising from neural network architectures. Each circle in the compu- lation graphs represents neuron or component in the architecture. Fach line segment represents one set of parameters. (a) Scaling of the incoming weights of an invariant activation (Examples 2.6, 2.7). (b) Scaling of the incoming and outgoing weights of an equivariant activation (Examples 2.2-2.5). (c) Permuta- tion of neurons (Example 2.8) or components (Examples 2.9) logether with their associaled weightss
Example 2.6 (Batch Normalization (lotie & Szegedy 2015 Batchnorm stondarlizes imputs of a layer
across mini-batch: BN(Z) 2 2-E/Z where the expectation and variance are computed row-wise Z. Varlz''
Assuming Z is a lineor feature Z = WX where W E = Rmxx and X E [ then the batchnorm function has positive scaling symmeetry. Batchnorm is equivariant with respect to the group R which consists of diagonal matriz with positive diagonal entries, acting on θ by W = gW, for E Ryo (Kunin et al. 2021). Other normalization layers, such as layer normalization [B el al. 2016), group normalization (Wu 8 He 2018), and weight normalization (Salimans Kingma 2016), often have scaling symeeetry of similar forms.
Example 2.7 (Softmax (Bridle 1989)). The softmar function a: R² + R is defined as o(2); e / e Consider the function f(W,X) (), where W θ Raxx and X E RRnkk and applies on the rows of WX. Then has a translation symmetry, acting on by (g W); W; g for each row W4, with 9 (R", +), the additive group over I. [Kunin et al. 2021)
Many architectures have parameter symmetries given by a symmetric group. This symmetry can often be interpreted as permutation of neurons or components along with their associated weights, as shown in Examples 2.7, 2.8 and Figure c). Similar forms of symmetry have also been identified in recurrent neural networks iAlerttiii et al. 1995).
Example 2.8 (Pointwise Activation). Feedforward networks with pointwise activations that use the same scalar function across coordinates have permutation symmetries. Consider a two-layer neural network J(W2, W1, X) W2a(W,X) X) with (W2, Wi) E activation function a with 0, Jj for all i,j. There is a permutation symmetry acting on θ by 9 (W2, Wi) WWaag"..Wii for g E Sn. [Heectt-ielser) 1990).
RRuxh Rhxx X Taaxk and any pointwise
Example 2.9 (Radial basis function networks Broomhhedd & Lowe 1988)). In radial basis function (RBF) [x-o_] with radial function yp: R4 + R and parameters (w,, b,, ci) E e = (R R_ x )² there is a permulation symmetry acting on O by क ((ws, bi ci)tm)) = (wx-1(0 by ( (o)) (7-))) for
network Σi= wis
a E Sy Kūrková & Nerudo 1994).
Viewed computational graph, neural network inherits the symmetries of all its subnetworks (Zhao et al. 2024a). Examples 2.2 2.9 therefore naturally exlend to multi-layer networks by applying symmetry to any subset of adjacent layer pairs. Additionally, modern architectures are often constructed from smaller components, many of which have the same form as these examples. Consoquently, they admit the same symmetries, which act on subspace of their parameter space.
2.2.2 Example: Symmetries in Transformers
To illustrate how symmetries emerge and combine in popular architectures, delineate the symmetries in transformers by examining their components (Zhang et al. 2025 Figure visualizes the source of these symmetries.
5

---

Published in Transactions on Machine Learning Research (12/2025)
Table 1: Symmetry in common neural network components (Examples 2.2 2.9 Since larger networks inherit symmetrics from their subnctworks, these examples naturally extend to multi-layer networks by applying symmelry to pairs of adjacent. layers.
Name
Architecture
Symmetry Group
Group action
Linear
W2W1X
GL, (R)
9 (W2, Wi) (W29* 9W
Homogeneous
W hom (WiX)
R
9 (W2, Wi) (W2g 9Wi)
Tanh
W20 (W,X)
9 (W2, Wi) = (W29 9WW)
Radial rescaling
W2rredial (W1X)
O(h)
g (W2, W1) (W29 (WW1
(W -WXX)
Batchnorm
g. W gW
v Var[(W
Softmax
softmax X)
(IR", +)
(g - W); = W; +9
Pointwise
W2 pointwiise (W
9 (W2, W,) = (W29** -1 WW1
Radial basis function E wipp
|
Sp
π (wy, bi, 04) = (w-- Un 10)) CT ((1)
b;
Example 2.10 (Attention Vaswani et al. 2017 The self attention function is core component of transformers. For key K Rmxp, query Q guxp, and value V E BRuxr
(QWCKWWYY) dde
Allention QQQ KW vWV) = softmax
VW
with imput embeddings (Q, V) pdxdm. x Rdxdm dRd and weights (Wq. WK. K, Wv) E Q =pmm xdu Idd xde Rdmxdd This architecture has a GLa, (RR symmelry, acting on O by g (WQ, WK NrrK,, (W²g) w gT. W for g E GLaz (R).
multi-head attention replicates the attention block across h heads and lin- early mixes their outputs, thereby inheriting the (GLa, (R)) symmelry and introducing additional head- permutation (Sh) and per-head linear symmetries ((GLd, (z))*).
Building on Example 2.10
Example 2.11 (Multi-head attention (Vawwan et al. 2017)). Most transformer architectures use multi- hend allentions, which concatenate the oulput of muullpple allentions and apply linear transforma- tion to the concatenated output. Concretely, malti-head attention is given by Multillead(Q, K, V) Concat(head,, headh) W with head, Attention QW, KKWK, VWV) and additional parameters Wo e Tk The new porameter space. is θ (Rdmxd) xde x Rddmxd x Tidm xd. Rhhd, xam. To simplify nota- tion when describing symmetries, denote W E xdd. the matrix formed by row (d, x xi+1) to row (do x (1) of W
In addition to the (GLa, (R))* symmetry inherited from individual attention heads, a malti-head atten- tion admits an Sn and a (GLa, (R))) symmetry. The Sp. symmetry acts as permutation of attention heads: π (W,?, W.K, W.Y wP) (w 1()) WK " WY w--1()) for E Sh. The multiplication of each attention head with Wo results in one GLd, (RR) symmetry, acting by gi (W. w.k. WV, w.) = (W,2, W.K 9,' ".,'',,.....', for 9, GLa, (R), i == h,...h. h.
Parameters in a transformer can be finetuned after training to adapt to new datascts or tasks. A commonly used method, low-rank adaptation (LoRA) (Hu et al. 2022), updates pretrained parameters by low-rank matrices to reduce memory and compute. This parametrization adopts a general linear symmetry group.
Example 2.12 (LoRA). In low-rank adaptation W + UV, with pretrained weight matrix W Raxm and trainable parameters U Rexr V Rrxm there GL, (R) symmetry, acting on (U,V) by g (U.V) (Ug-). for g (R)) (Pulterman et al. 2024

---

Published in Transactions on Machine Learning Research (12/2025)
(a)
(b)
6
Sn symmetry: (Wop May Way Wo.) (Worsso Meg Wong W Ma
softmax
VWy
0
(Glu, (RO)** symmetry:
Matrix Multiplication
Bi (Wo, Wo.) (Wr,9i 9/
Matrix Multiplication
(WWOWER
5o0tmax
sodtnaan- JKW,
softiaed-
cottmia (WWW,
SoftMax Scale
Concatenation
GL4, symmetry:
(Wo. We)
softmaa JPW
(Wog"* "or""""WWW"" Wya')
(GLE (R))
Matrix Multiplication
Attention
Attention
Attention
symmetry
Parameters:
W Q
Wx
Wy
We, Ws. Wr
We
Wxz
Woz
Web Wa,
Wra
Wo
Data:
K.V
Figure 4: Symmetries in (a) the attention mechanism and (b) a multi-headed attention. Dashed rectangles represent output of cach layer. Each symmetry annotated in the same color that of the component that gives rise to it.
2.3 Relaxed Definitions of Symmetry
While functional neural network symmetries ensure invariance of the neural network output across all input data, more general notions of symmetry arise when we relax this requirement. In this section, we expand the scope of parameter space symmetry to include transformations that preserve the overall loss function instead of the neural network function, as well as transformations that preserve the out over subsets of data instead of the entire data space.
2.3.1 Loss Symmetry
Loss symmetry refers to parameter transformations that preserve the overall loss but not necessarily the neural network function. Recall that the loss function L: θ x D + IR is often the composition of model f: Piuput + Ptarget and a cost function e: Dearget X Dtarget + R. Definition 2.4 defines transformations on that do not alter the value of f, thereby also preserving L. We now relax this definition to include transformations that are required preserve L but allowed change f.
Definition 2.6 (Functional loss symmetry). Let θ be the parameter space of loss function L: 1:0x7 R. A parameter space symmetry of is an action of group G θ that leaves L invariant:
L(g 0,x) = L(0,)), E G, E O, Vr E D.
Example 2.13 (Self-supervised learning with linear network Ziyin et al. 2023b ). Consider a self supercised learning setting, with linear network f(W, x) = Wx and loss L(W, x) that depends on f only through f(x) for data pairs I, I E R² Then, L has a rotational symmetry, which acts on θ = pmmx by g P-W =gW for in ((m). This differs from the group action on linear networks (Example 2.2 since only aets on one layer and not canceled by trunsformations in other layers. Hence, this group action preserves the overall loss L but not the feedforward function.
2.3.2 Data- Dependett Symmetry
In the next definition, we relax the definition of functional symmetrics (Definitions 2.4 and 2.6 by consid- ering parameter transformations that preserve the output for data batches of size n, without guaranteeing invariance on other data. These symmetrics depend on the data with respect which the neural network output invariant.
Definition 2.7 (Data-dependent group action). A data- detendennndnntt group action is a group action of G on ( D that acts trivially D To simplify notation, drop Dr. from the oulput and wrile dala-

---

Published in Transactions on Machine Learning Research (12/2025)
dependent group action map x (8 D") - θ that satisfies (0, = and g (g' (0, X). (g9'). (0, X) for all 9.96 in G, 0 in 6 and X E D
Definition 2.8 (Data-dependent symmetry). Let θ be the parameter space. Let F: Ox Y be a function with output space Y. A dato-dependent symmetry of F is data-dependent action of group G on O that preserves the value of f on subset of data:
(0,x)))) 2(0,2), E G, 40 O, VX E (Dinpu)", and Yx X.
Example 2.14 (Two-layer network Zhao et al. 2023
For a nonzero vector E R", define matrix R as
2s cos((a)-1)
[I sin(ar))
if S and II sin((r)) +
(R2) =
-rs)n(a;)
if j = otwwissee.
Consider bwo-layer neural network f(W2, W1, X) W2I(WiX), where (W2, Wi) E e Ruxh Rbxn and X E R". Suppose (2) is nonzero for any E IR.h Then this archilecture has dala-dependenl GT&(R) symmetry, which acts ④ by g (W2, W1, (o gW1) and preserves loss value single data points.
2.3.3 Distribution Symmetry
In practical settings, data can often be viewed as samples from an underlying distribution. The following definition considers parameter transformations that preserve the expected loss over this distribution. Neu- ral networks related by these transformations are expected to perform similarly on data within the given distribution but not guarantoed to perform similarly on data from different distributions.
Definition 2.9 (Distribution symmetry). Consider a function F: θ x + with output space Y, where θ is the parameler spuce and dala are drawn from a distribution D. A distribution symmelry of is an action of group G @ that leaves the expectation of F invariant:
EF=D[F(g 0, x)] = EF=D[F((,x)) E VOE
Distribution symmetry is often related to averaging effects, where aggregated predictions remain consistent despite variations in the predictions from individual parameters.
Example 2.15. Consider the function F; mx x (R" x R") + R. defined by F(W, v)) = Wr where W is the parameter and x, are data. Assume that x is drown from a distribution D2 and for each data pair (I, y), y is defined as y = W for a fized matriz w E max, Let i = [z]., Then, the expectation of under the data distribution
EgND, [FWW, (a, y))l = ELND, Wr W WE W"
Let G be the group of all invertible matrices A for which Ax = x. This group acts on the parameter space maxn via the action A W = - This action a distribution symmetry for F, because for all A G and matria W E Rmxx we have:
EAED, [F(A W, (z, v))l = Eg W A W' WJ W = Rz (FW.
The relation among the above definitions are visualized in Figure 5 Parameter transformations that preserve the neural network function are guaranteed to preserve the loss function. Parameter transformations that preserve a function at every data point are guaranteed to preserve the function over every fixed-size batch or distribution. Therefore, neural network symmetry (Definition [.4] implics loss symmetry (Definition 2.6 Neural network (or loss) symmetry implies both neural network (or loss) data-dependent symmetry (Definition 2.8 and neural network (or loss) distribution loss symmetry (Definition 2.9

---

Published in Transactions on Machine Learning Research (12/2025)
(a)
(b)
(funtiooallctt---epeeeee///iiori loss symmetry
Functional (loss/NN) symmetry [Definition 2.6]
[Definition 2.6]
(functoonn/ddaaadeeeeddd///fffio depenndent/distriutttonn neural network symmetry
Data dependent (loss/NN) symmetry [Definition 2.8]
Distribution (loss/NN) symmetry [Definition 2.9]
[Definition 2.4]
Figure 5: Relation among different definitions of parameter space symmetry. Arrows represent the relation- ship that satisfying one definition implies satisfying the other. (a) Symmetries that preserve the overall loss versus the neural network output. (b) Symmetries defined over the entire input space versus those defined over subsets of data.
2.4 Parameter Identifiability and Completeness of Symmetry
Having introduced a range of examples highlighting that symmetry is ubiquitous neural network param- eter spaces, we now turn to the fundamental question of whether known paramcter symmetrics account for all the ways in which parameters can represent the same function. This question concerns parameter identifiability- the extent to which a neural network's function determines its parameters, which is important in interpreting and comparing trained networks (Ran & Hu 2017). This section formalizes the relationship between symmetry and identifiability using the realization map from parameters to functions, and surveys results on when known symmetries are sufficient. to characterize all such redundancy.
Formally, we consider the realization map p: θ + F, where == Dimput + Doutput which associates each parameter 0 E θ with the function p(0) F it defines (Grigsby al. 2025). The fiber of a function f under P; denoted o (ff := {0 E p(0) = f), is the of all parameters that map to the same function f. In many architectures, P is not injective, and fibers contain multiple distinct parameters that define the same function. This implies that parameters are, in general, not uniquely determined by the function alone.
Parameter space symmetries contribute directly to this non-identiffabiility, as they induce transformations within the same fiber. Therefore, analyzing fiber structures helps in determining whether the known set of symmetries is complete. If symmetry group G acts on θ such that every element of a fiber can be reached from another via a transformation in G, we say G acts transitively on that fiber. In this case, G captures all the redundancy in the parameterization, and no hidden symmetries remain.
Definition 2.10 (Parameter Identifiabilityy. Let L: xx R be loss function with symmetry group G. We say parameters of L are identifiable up to G if for every fiber of the realization map of L, the group G acts transitively on that fiber. We say G is complete if parometers are identifiable up to G.
For several architectures, such completeness has been established. For example, Gaussian radial basis func- tion networks are known to be identifiable up to permutation symmetry, indicating that permutation sym- metry complete for these architectures (Kūrková & Neruda 1994). For feedforward networks with tanh activations, if two networks represent the same function, then they have the same architecture and parame- ters up to permutation and sign-flip symmetries, under known set of conditions (Sussmann) 1992 Fefferman & Markel 1993). Similar completeness results have also been established for complex-valued tanh networks NNitaa 2003 Kobayashi 2010), (anh recurrent neural nelworks (Kimura) (2022), networks with aclivation functions a satisfying (0) = 0, (0) * O, and o(()) = 0 (Albertini & Sontag 1993), and networks with asymptotically constant activation functions (Kúrková & Kainen] (194)). For ceathrreoff-xpperee (MoE)
architectures, iitiutaatioottanllationn symmetriie of the gating parameters were recently proven to fully
characterize functional equivalence (Tran et al. 2025b)

---

Published in Transactions on Machine Learning Research (12/2025)
The identifiability of ReLU networks is less straightforward. In particular, ReLU networks can contain hidden symmetries-ttanssornne transformations that preserve the realized function but not captured by permutation and positive scaling symmetries. Grigsby et al. (2023) describe several mechanisms through which hidden symmetrics can arise. For example, for a single ReLU neuron with scalar input I and realization map p: (a, b) + (x + ReL.U(az+b), the fiber of the constant function 0. o (0), contains (0, 0) and (0,-1), which are not related by rescaling and permutation (Grigsby et al. 2025). To formalize this variability, Grrigsby let al. (2025) introduce the functional dimension, defined as the rank of the Jacobian of the realization map P, which reflects the local dimensionality of the set of functions realized near a given parameter. They show that the functional dimension may not be constant even within the same fiber. This suggest complicated structure of symmetries in ReLU networks.
Other recent work investigates when ReLU networks are identifiable up to positive scaling and permutation symmetries. (Petzka et al.) 2020) prove that excluding a degenerate case where two neurons have identical zero hyperplanes, two-layer ReLU networks do not have additional symmetrics besides permutation and positive scaling. (Rolnick& & Kording 2020) prove that under the assumption that networks within each linear region compute the same linear function, boundaries between linear regions uniquely determine parameters up to permutation and positive scaling. For ReLU networks with cither non-incrcasing widths (Bui Thi Mai & Lampert 2020) or all layers wider than input (Grigsby et al. 2023), a positive measure subset of the parameters have no symmetries besides permutation and positive scaling (hidden symmetries). Grigsby iet al. (2023) also empirically show that the probability of having no hidden symmetry increases with input and layer width and decreases with depth. Pola--eelisseer et al. 2023) provide sufficient conditions under which two ReLU network are identical up permutation and positive scaling if their output agrees subset of the input space.
The identifiability of neural networks that are polynomial in their parameters can often be analyzed using tools from algebraic geometry (Marchetti et al. 2025). A notable example is multi-layer perceptrons with polynomial activation functions, or polynomial neural networks. Kilcel et al. (2019) conjecture that permuta- tion and scaling are the only symmetries in generic polynomial neural nelworks, which was later proved for the monomial case (Finkel et al./ 2025). For more general polynomial activation functions, there are only finitely many data-independent parameter symmetries Shahverdi al. (2025) Convolutional neural networks with polynomial activations generically have no nontrivial data-independent parameter symmetries. Another ex- ample of a polynomial function is lightening sell-attention (W², W WW) + QW TVWV, yVW where the softmax normalization is omitted from the standard attention mechanism (Example 20 Henry et al. (2025) prove that in networks composed of lightming self- attention layers, the only symmetrics are (1) scaling WoWk K and Wy by a constant, (2) scaling Wq and WK by an invertible matrix, and (3) scaling Wy of one layer and Wq. Wx in the next layer by an invertible matrix. They conjecture and verify numerically that adding back the softmax breaks only the first of these symmetries.
3 Role of Symmetry in Loss Landscapes
The loss landscape- graph of the loss function over the parameter spaco-plays a central role in un- derstanding the optimization behavior of neural networks. Parameter symmetries and the lack of param- eter identifiability increase the size of loss level sets. To illustrate, consider two-layer linear network ftinear (W2, Wi) W,W,,X, where (W2, Wi) E θ = m x Tg and X E Baxk is fixed input data. If (Wa, Wi) is a global minimum, so are all points (Wig "WWT) with E (RR. This set forms a positive-dimensional manifold of zero loss (Figure G
In this section, we examine how parameter space symmetry governs the connectedness, dimensionality, and structure of loss level sets. In Sections and 5 we will see how these properties can be exploited in optimization.
3.1 Continuous Symmetry and Mode Connectivity
Continuous parameter space symmetries enlarge individual minima into connected sets. Formally, a sym- metry is continuous if the symmetry group G is Lie group and the group action is continuous. The orbit
10

---

Published in Transactions on Machine Learning Research (12/2025)
level sets
0
0.8
LVV. 1, W/2
0.4
0.0
2
3
3
2
W2
1
0
Figure 6: Parameter symmetries increase the size of the zero loss set. If 0 is a global minimum, are all points 0 with in the symmetry group of L. This set is typically positive-dimensionall
of point O the set. 0 (0' e 0' = g. for some g E G}. When G Lie group, the orbits are often positive-dimensione so continuous symmetries generically create positive-dimensional manifold lying entirely in a single loss level set.
Empirical work first revealed that independently trained networks can be joined by low-loss curves in parame- ter space (Garipov et al. 2018 Draxler et al. 2018). This observation, often referred to as mode connectivity (Garipov et al. 2018), has been exlended to show that points found by stochastic gradient descent (SGD) are connected by dimeti-dimensional volumes (Benton et al. 2021). This insight has applications in model
Benton al. 2021 Benzing al., 2022) and model averaging (Izmailov et al. 2018 Wortsman et 2022 Continuous symmetries provide theoretical explanation for this phe-
ensembling Gariipvv al. 2018
nomenon. The loss remains constant along each orbit, so minima discovered by SGD will automatically be connected whenever the group action is continuous and they be mapped to each other by element in the identity component of the group. We note that not all points in minimum are related by a symmetry transformation, and empirical connectivity observed between independently trained networks may also arise from other mechanisms beyond symmetry.
Parameter space symmetry imposes structure on the loss level sets. The minimum of a neural network, which is the optimization target, is perhaps the most interesting level set. Investigating the connection between symmetry groups and the topology of minima may lead to better understanding of the loss landscape and explain the source of mode connectivity. When a level set is homeomorphic to the symmetry group through a group action, e.g. in linear networks with invertible weight matrices, the minima has the same topological properties, such as connectedness, as the symmetry group (Zhao et al.) 2025). Zero-loss curves generated by contimuous actions (Zhao et al. 2024bb and dimension bounds dorived from orbit dimensions (Zhao el al. ) further illustrate how symmetry dictales flat directions. (Lengyel el al. 2020) visualizes the set of functionally equivalent parameters. In ReLU networks, the visualization demonstrates the expected permutation and scaling symmetry, well other structures which indicate complex symmetrics.
3.2 Discrete Symmetry and Structure of Minimum
Discrete symmetries do not connect minima continuously; instead they replicate them, creating many func- tionally identical copies throughout the parameter space. Permutation is the best-studied discrete symmetry.
11

---

Published in Transactions on Machine Learning Research (12/2025)
Brea et al. (2019) show that all permutations of given hidden layer reside in the same loss level set, while Simgek et al. (2021) prove that adding just one extra neuron per layer in minimal tanh network merges
these replicas into a single connected manifold. Extending this analysis, Fobbegia-Roberts (2023) charac-
terize functional equivalence classes for hyperbolii-tangent networks, and Pittorino et al. (0022 examine
minima in the function space after quotienting out both permulation and scaling symmetries. Counting arguments reveal the combinatorial explosion such symmetries induce. Michelucci (2022) estimate that the number of permutation-equivaal complicates landscape exploration.
minima grows factorially with width, underscoring how discrete symmetry
Although discrete actions do not generate continuous valleys, they still influence the loss landscape shape and optimization trajectories, as the next section will demonstrate. In particular, quotienting out these symmetries reduces model redundancy, shrinks the search space, and sometimes simplify sampling.
3.3 Removing Symmetry: Model Compression and Reduced Search Space
While symmetry enriches the loss landscape by introducing structure and multiplicity, removing symmetry reveals a simpler goometry-one that is more compact to represent, and often more amenable to analysis. Symmetry removal effectively collapses each symmetry orbit to a canonical representative (Sorensen 2020). This operation alters the shape of the loss landscape-nol by changing its values, but by restricting attention to lower-dinensional quotient space where each point represents a unique function. As result, the loss landscape becomes less redundant, more compact, and casier to sample from and optimize.
One major motivation for removing symmetry is to reduce the dimensionality of the parameter space without altering the function space. This operation, often known as model compression, is of critical importance in lowering storage cost, improving inference spoed, and efficient deployment of machine lcarning models. Every symmetry implies a set. of directions in which the loss is constant, stretching level sets into high- dimensional manifolds. Quotienting out these directions yields more compact version of the loss landscape, where each point corresponds to unique function. In the context of model compression, this means can represent networks more efficiently. For example, Ganev et al. (2022) compress radial neural networks by factoring out orthogonal symmetries, and Sourek et al. (2021) exploit structural symmetries in computational graphs to merge redundant nodes. These methods reduce storage costs and inference time, without affecting expressivity or accuracy.
Beyond compression, removing symmetry transforms the loss landscape by collapsing symmetry-related regions into single representatives. For networks with permutation symmctry- as feedforward networks with interchangeable hidden units- the parameter space can be partitioned into large equivalence classes. Building on the observation that permutations of neurons can be viewed as compositions of reflections in the parameter space, Hecht-Niclson (1990) constructs a cone in the paramcter space that contains a permuted copy of every point in the parameter space. For two layer networks with permutation symmetry Sn, the cone occupies only 1/h of the parameter space. They further prove the existence of minimal search set, in which no two points are related by symmetry, This geometric reduction reduces the volume of minima and reveals its global structure. Lim et al. (2024b) empirically show that after removing permutation and scaling symmetries, neural networks are more linearly mode connected, and the loss decreases more monotonically on the linear interpolation between initialization and trained parameters.
In Bayesian neural networks, removing symmetry reduces the effective search space by collapsing function- ally equivalent. modes in the posterior (Lim et al., (024bb). In high-dimensional models like neural networks, symmetrics can cause postcrior distributions to bocome multi-modal in a functionally redundant way. This impairs MCMC mixing and complicates interpretation. By mapping parameters to canonical representa- tive- effe-ctvcy quotienting out permutation or sign-flip symmetries- -eiee
Wiese et al. (2023) and Laurent et al. 2024h show that it is possible to explore more informative and diverse set of functionally distinct solutions. Xiao et al. 2023) show that removing permutation symmetry leads to a compact representation that helps directly compare trained Bayesian neural networks across sampling methods.
Finally, removing symmetry alters the curvature and critical point structure of the landscape, with direct implications for optimization. Symmetry-induee flat directions often give rise to plateaus or saddle points that slow down training. By projecting the loss landscape onto a symmetry space, these degenerate
12

---

Published in Transactions on Machine Learning Research (12/2025)
directions are removed, yielding more strongly convex surface. Leake & Vishnoi 2021 show that continuous symmetries can be used to construct convex polytopes, allowing nonconvex optimization problems to be reformulated as linear optimization problems over polytopes, which are computationally more tractable.
In summary, removing symmetry simplifies the geometry and topology of the loss landscape, by collapsing orbits of equivalent parameters. This simplification yields a lover-dimensional loss surface, a more meaningful proba bilistie structure in Bayesian neural networks, and an optimization landscape with fewer spurious critical points and degenerate directions. While symmetry endows the loss landscape with structure and multiplicity, its removal distills the parameter space into representation more aligned with the underlying function space.
4 Applications of Symmetry in Gradient-Based Optimization
In this section, we describe ways of leveraging parameter space symmetry to design more efficient and thoorctically motivated optimization algorithms. Optimization in machine learning is the process of finding parameters that minimize an objective function L: θ x D + R, over a given subset of data S D. In the absence of additional constraints the parameters, the optimization problem is formulated as
min X).
A common optimization method is gradient descent, which iteratively updates parameters in the direction opposite to the gradient (0) aL
'0 Using step size tpp E Rso, the parameter after t 1 steps is obtained
from the paramcter after steps by
0++1= = 6, VI.)).
When analyzing gradient descent, the presence of parameter symmetry roduces complexities that are not immediately apparent from the loss values alone. Consider two points in the parameter space, 0 and 0' = g for some g in the symmetry group G. While the value of at and 0' is the same, the gradient VL may differ (Figure 7p). Therefore, learning dynamics starting from points in the same orbit can also be different
Van Laarhoven
2017 Tanaka & Kunin 2021 (Figure 7b).
The discrepancy among points in an orbit inspires two families of optimization techniques purposefully searching for a favorable point within the orbit, or eliminating the differences among points in the same orbit. We discuss the two approaches in the next two subsections.
4.1 Exploiting Difference among Points an Orbit
Leveraging the difference among points in an orbit, loss-invariant transformations defined by symmetry may be used in concert with common optimization algorithms to accelerate training or improve the quality of the final solution. During optimization, the parameter 0 can be transformed by an element 9 of the symmetry group G, 0 + 9 (Figure 7 We can thus optimize over the orbits by optimizing over the group G. If G is Lie group, then optimization can likewise be performed using gradient descent on the manifold underlying G. Below we discuss various optimization objectives for g, including minimizing parameter norms, maximizing loss gradients, and optimizing other qualities the solution (Table 2
One application minimizing the norm of the parameters, which has been shown to improve learning. On ReLU networks, [Sock et al. (2019) and Saul (2023) search in the positive rescaling group for an element that minimizes parameter norms. Stock et al. (2019) propose Fquui-normalization (FNorm),
algorithm that rescales parameters 0 to minimize their 4 norm (,(0) = -101 Saul (2023) proposes to minimize
the norm [Wllp.s = (E, (E ))),,")) where W.j denotes the weight from neurons i to j. In the resulting network, the incoming and outgoing weights are balanced at each layer. Interleaving these symnetry transformations with stochastic gradient descent improves training speed and test accuracy, following the intuition that gradient propagation smoother on parameters with smaller norms.
Other work seeks to improve convergence speed by optimizing on the orbits to increase the norm of gradients. On general neural network architectures, Armenta et. al. 2023) propose neural teleportation, which moves
13

---

Published in Transactions on Machine Learning Research (12/2025)
(a)
(b)
VL
θ
g
L(
0
01
91
(c)
θ
9 9
θ
A(0, t)
Alg 0, 2)
g A(O, t)
01
0
Figure 7: Implications and applications of parameter space symmetry in gradient-based optimization. (a)
Points in orbit have same loss values L but different gradients V L. (b) Gradient descent trajectories originating from two points in the same orbit, 0 and 0, may differ. (c) Algorithms leveraging these differences search within the orbit for a point that leads to faster optimization (g" -0. (d) G-invariant
algorithm A, which commutes with the group action, is indifferent to initializations within an orbit.
parameters on their loss level set using a group element random drawn from a distribution on the symmetry group. This procedure leads to an increase in the expected magnitude of gradients, resulting in improved empirical convergence rates. Building on this work, Zhao et al. (2022) perform optimization in the orbit to find points with large gradient norms ||V/]|2 They show that points where gradient norms are maximized in loss level set, the gradient descent direction VI. aligns with Newton's 's direction H-IVL, where H is the Hessian of L. This alignment suggests that optimal teleportation before each gradicnt descent step enables the algorithı to achieve the faster convergence rates lypical of second-order methods. In preconditioned gradient descent 01+1 0g - nAVL((;) where A is matrix with dimensions matching θ, teleporting to maximize the Mahalanobis norm ||V] results in faster convergence compared to maximizing the 12-norm ||L[|2 (Zhao et al.] 2024b Since introduction, teleportation has found broad application in long- tail recognition {Vang al., 2025). continual optimization in multi-task learning (Zhou et. al. 2025). and verification of deep learning inference (Maheri et al. 2025).
Beyond finding good starting point for gradient descent, optimization orbits incorporate other objectives at different times during training. For example, 7hhoo et al. (2024b) teleport parameters to regions with different sharpness and curvature, empirically improving the generalization ability of the final solution. More generally, knowing the symmetries allows one to search on the minima for points with desired properties. When the optimization on orbits done using infinitesimal symmetries, this approach similar orthogonal gradient doscent in continual learning (Farajtabar et al. 2020), which projects the gradient of new lasks to the space orthogonal to the gradient of the original task to avoid catastrophic forgetting.
Group actions that preserve loss approximately but not exactly are still useful tool to explore near loss level set. In this case, one can leverage the different loss or functions realized by points in the same orbit. In certain methods with regularizers, loss is not invariant on the orbits, but the variation is small, which slows down optimization after parameters enter the orbit. This is analogous to the Goldstone mode in physics (Altland & Simons 2010). After optimization on the full parameter space slows down due entering the Goldstone mode, Bamler & Mandt (2018) propose optimize on the symmetry subspace, which accelerates optimization in these orbits with beokly- symmetry.
14

---

Published in Transactions on Machine Learning Research (12/2025)
Table 2: Algorithms utilizing group elements to transform parameters during optimization.
Algorithm
Reference
Group Element for Parameter Update
ENorm
Stock et al. (0199)
arg mingee l1g 011
arg min EG ( (E W) 1))
Weight Balancing
Saul (2023)
Armenta et al.
2023
sampled from distribution on G' arg maXgeG (I
Teleportation
Zhao et al. [0022]
Zhao et a.l
2024b
arg max Curvature( 0)
Goldstonc-GI
Bamler & Mandt (0018
arg migminec geG L(g 8)
4.2
Symmetry Invariant Optimization Algorithms
Points in a symmetry group orbit share the same loss but can differ in other aspects. In Section 4.1 we explored optimization algorithms that capitalize on these differences. complementary approach of handling these differences involves designing learning algorithms that remain unaffected by such differences. In this section, discuss ways to make gradient descent invariant to parameter symmetries.
Similar to symmetry in the loss function, symmetry in algorithms is defined by the invariance of an algorithm to group actions on parameters. We denote an algorithm as A: x R - θ, which takes initial parameters with a dataset and outputs the learned parameters after steps. Inspired by the definition of equivaraa learning algorithms (Abbe & Boix-Adscra, 2022), the following definition generalizes rescaling invariant algorithms (Neyshabur al. (2115) to general aerrammeter-quvvriiantt algorithms, visualized in Figure Zi. Definition 4.1 (G-invariant algorithu). Let G be symmelry group of loss function L: Q D - R. An algorithm A is G-invariant i,, for every g E G, 0 E O, C D, and 1 >> 0, there erists an g E G such that A(g 0, X, t) = g' X,))
4.2.1 Scaling Invariant Algorithms
ReLU networks are positively rescale-invarianl (Example 2.3/, but gradient descent. (Equation 1 on them is not. When parameters are rescaled in ways such that there are significant disparities between imput and output weights of ncurons, gradient descont tends to perform poorly (Ncyshabur et al., 2015). The extra effort required to circumvent these configurations, along with difficulty in analysis caused by inconsistent performance across parameters on the same orbit, motivates new optimization algorithms that are invariant rescaling.
Two families of rescale invariant algorithms have been developed. The first one is based on the observation that the product of incoming and outgoing parameters of ReLU function is invariant to rescaling. From the parameters 0 of ReLU network, Neyshabur et al., (0015) construet a path vector (0) that contains the product of parameters along each path from an input node to an output node. Their algorithm, Path-SGD, performs proximal gradient descent with respect to a path regularizer [|=((0) - x )l instead the 2-norm
of parameters ||0 11000'10 00112 that would give rise Equation i
1 g++1 = argminn (VL())))) 6) + || (((0))nm) -()0))-(('))1 (B8))
0
The resulting update step size for each parameter is inversely proportional to the norm of a vector, where each entry is the product of all other parameters along a path that includes the parameter. Consequently, parameters with larger values receive proportionally larger updates, and the resulting algorithm achieves rescaling invariance.
Another method make gradient descent invariant to scaling symmetry involves projecting parameters to different space, then either optimizing directly in that space or, when the space is not vector space,
15

---

Published in Transactions on Machine Learning Research (12/2025)
performing manifold optimization. The space is often quotient space induced by scaling equivalence and has fewer dimensions than O Meng et al. 2019) design a rescaling invariant algorithm, G-SGD, by performing gradient descent in the rescale-invariatt space spanned by the path vector of basis paths. Badrinarayanaa
ict al. 2015) and Huang et al.) (2020) constrain the incoming weights of cach neuron to stay unit-norm by updating the weight. matrix W of each layer using the Riemannian gradient before projecting them back to 7}. In addition ReLU networks, other architectures such as those with batch normalization also exhibit scaling symmetry (Example 2.6), and similar optimization techniques apply. Yi (2022) quotients out the positive scaling symmetry in batch normalization, develops gradient descent a algorithms on the quotient manifold in a similar way, and proves that the resulting algorithm has a better convergence rate than gradient descent in the original parameter space.
the oblique manifold (W E [R=*P: diag(WWT)
4.2.2 General Symmetry Invariant Algorithms
One algorithm whose convergence is invariant to general parameter symmerry is natural gradient descent. This method computes gradients using the Fisher metric on the manifold of distributions, which is invariant to parametrizations (Amari 1998). In implementation, natural gradient descent is not fully invariant to symmetry due to finite step size. However, methods which use a second-order ODE solver and the second- order approximation of the exponential map can reduce the invariance error to second order (Song et al., 2018). More recently, Kristiadi et al. (2023) observe that gradient descent can be made invariant to paramcter symmetry by explicitly including the metric when computing gradients, and show that there exist metrics other than the Fisher metric that achieve this invariance. Scale-free updates, which are updates invariant to gradient scaling, are hypothesized to be the reason why AdamW outperforms standard Adam with & regularization ZZuang el al. 2022
5 Implications of Symmetry for Learning Dynamics
In this section we continue to explore how parameter space symmetry informs learning dynamics. In physical systems, Noether's theorem states that certain continuous symmetries have associated conservation laws (Noether) 1918). Similarly, in neural network training, parameter space symmetry implies the existence of conserved quantities in gradient flows. These conserved quantities, which remain constant along gradient flow trajectories, are useful both as conditions held throughout training and as a way to parameterize optimization trajectories in the parameter space. We summarize known conserved quantities arising from parameter space symmeetry and discuss their roles in theoretical analysis of learning dynamics.
5.1 From Symmetry to Conserved Quantities
Gradient flow is a continuous version of gradient descent, commonly used to analyze learning dynamics in the limit of infinitesimal step sizes. It defines a trajectory ((t) E θ for t E R>o that connects an initialization to the limiting critical point, with velocity given by the gradient of loss: O(t) - Veay L. A conserved quantity of gradient flow is a function Q: θ R that remains constant along this trajectory, i.e., Q(0(s)) = Q()())) for all 8, I Rgo-
A widely studied conserved quantity in gradient flow is imbalance- the difference between the Gram matrices of adjacent layers in linear networks.
Example 5.1 (Imbalance (Arora et al. 2018b)). Consider an l-layer linear feedforword network f: θ x D Dinput > Diaeget, defined as (Wi,.., Wi W + W,,.W, with parameter space θ = RexxX=-1 .... X Xoo Define the loss function L:
as the composition of f and any differentiable function w(h+1)
(ep(++)yyT(++)) E X is C: Draryet x D Draryet R. For h E [1 1), each element in W ((o0))---(((+))))))) (W(++)) conserved quantily of the grudient flow on L, known as the unbalancedness Du el al. 2018) or imbalance
Turmoun et al. 2021)
Similar conserved quantities have been identified beyond linear feedforward networks and gradient flow. For example, in foodforward networks with homogencous activations, Duu et al. (2018) show that gradient flow preserves the difference in squared norms between each neuron's incoming and outgoing weights, making it conserved quantity. In graph attention networks with homogeneous activation functions, Mustala et al.
16

---

Published in Transactions on Machine Learning Research (12/2025)
(b)
= -3
=
3
VQ and symmetry direction
minimum
L(W1, W2)
VL
4
0.0
Loss level set
2
W2
Figure 8: (a) Symmetry and corresponding conserved quantities. Symmetry of parameter space symmetries and the gradients of conserved quantities, VQ; are both orthogonal to V and hence lic tangent to the loss level shown the figure. Symmetrics and conserved quantities can therefore be connected by matching VQ to symmetry direction. (b) Conserved quantities partially parameterize gradient flow trajectories and minima (adapted from Figure in Zhao et al. (2023)). Different trajectories correspond to distinct values of the conserved quantity Q, which remains fixed along the trajectory. The minima reached by these trajectories also have distinct conserved quantity values.
actions
(023) identify similar conserved quantities the difference between the norm of incoming and outgoing parameters of each neuron. By expressing convolutional layers as equivalent fully connected layers, Le & Jegelka (022)) identify conserved quantities of similar forms between convolutional layers and between residual blocks in ResNet. Beyond standard gradient descent, Huh (020) identify conserved quantities under spectral initialization natural gradient descent and curvature-corrected dynamics, including expressions involving differences and ratios of singular values of weight matrices.
Several theoretical frameworks explicitly connect conserved quantities parameter space symmetries, draw- ing analogies to Noether's theorem and extending conservation laws across diverse optimization dynamics. For example, Kunin et al.] (2021) derive conserved quantilies in gradient flows and analyze their evolution under modified gradient flows that better approximate stochastic gradient descent. Their work focuses on one-parameter symmetry groups, such as translation, scaling, and rescaling, and parallels Noether's thco- rem in both form and intuition. Extending these ideas to higher-dimensional symmnerr groups, Gluch & Urbanke (2021) develop a framework for describing optimization algorithms as ordinary differential equa- tions, identifying their corresponding Lagrangians, and deriving conserved quantities by applying Noether's theorem via the generators of symmetry groups. Notably, they connect imbalance to symmetry and derive conserved quantities for a broader class of dynamics, including both first-order and second-order systems such as Newtonian dynamics and Nesterov's accelerated flows.
For more general architectures, Zhao et al. 2023 associate conserved quantities with infinitesimal symme- tries, deriving them for activation functions with general equivariance properties. This approach recovers known conserved quantitics for homogencous activations and reveals new consorvation law analogous angular momentum conservation for radial rescaling activation layers.
While many conserved quantities question whether all conserved quantitics originate from known symmetries. To explore whether known sets of conserved quantities are maximal, Marcotte et al. (2023) apply the Frobenius theorem to compute the number of independent conserved quantities, from the Lie algebra generated by the vector fields spanned by the model's Jacobian. In the case of the matrix factorization problem (Example (222 without the bias terms), they show that besides imbalance, there are no other independent conservation laws. Similar results hold for attention layers in transformers (Marcotte al. 2025).
be derived from parameter space symmetries, it remains open
17

---

Published in Transactions on Machine Learning Research (12/2025)
5.2 Conserved Quantities for Convergence and Parameterization
By ensuring certain properties remain constant throughout training, conserved quantities provide stability in gradient flow that is essential for convergence analysis. In particular, any condition based solely conserved quantities that holds at initialization will continue to hold throughout the gradient flow. As noted in (Gluch & Urbanke 2021). conserved quantities such as imbalance provide guarantees that parameters are bounded throughout gradient flow, which is helpful in convergence analysis. The proofs of several convergence bounds for gradient descent of deep linear networks depend either on the invariance of the imbalance (Ji & Telgarsky, 2019) or the assumption that the imbalance is small or zero (Arora et al., 2018a b). Zero imbalance also an assumption in (Bah et al. 2022), as a condition under which the gradient flows of linear networks are Riemannian gradient flows on the manifold of fixed rank matrices.
Beyond stability, conserved quantities act coordinates for the dynamics. Because they are constant along a trajectory, they label optimization paths, and their limiting values locate endpoints within the set of minima. This labeling links initialization to both convergence and generalization: in small two-layer networks, Zhao et al. (2023) observe strong correlations between conserved quantities and both convergence and sharpness of the resulting minima, suggesting that carefully chosen initializations can improve training efficiency and generalization. Moreover, under imbalanced initializations, conserved quantities have been explicitly connected to the convergence rate by appearing terms in convergence bounds Tarmoun et al. 2021
Min et al. 2021 2023
Xu et al. 2023
By parametrizing minima, conserved quantitics also help describe which solutions optimization converges to, formalizing the notion of implicit bias - the tendency of optimization algorithms to favor solutions with specific properties. In homogeneous and leaky ReLU networks, Du et al. (2018) and Kou et al. (2023) prove that layers become automatically balancod. Similarly, Waan et al. (2022) proves an implicit regularization effect in matrix factorization problems under large learning rates. This effect is quantified by an upper bound the difference between the 2-norms of weight matrices, quantity similar to imbalance. This trend also appears empirically Kunin al. (2021) observe that imbalance decreases exponentially when training with large learning rates. For loss functions with a rescaling symmetry, (Ziyin et al. 2023a) derive the stationary distríbution of SGD and prove that SGD solutions are biased towards a balanced one, different from predictions from a Langevin model. Extending this idea, Ziyin (0024) propose a unified framework based on mirror reflection symmetry, encompassing rescaling, rolation, and permutation symmetries. They prove that such symmetries impose structured constraints, which are preferentially satisfied when weight decay or gradient noise is large. This framework explains various phenomena in gradient-based learning and motivates algorithms that enforce constraints.
As such, conserved quantities, rooted in symmetry, bridge the structure of parameter space and the dynamics of learning, guiding both the path and outcome of optimization.
6 Connections to Symmetry in Internal Representations and Data
In previous sections, we focused on how symmetry impacts the structure and dynamics within the parameter space. In this section, we explore how parameter space symmetry interacts with symmelry in other spaces, including data and internal representations. In the first part, we examine how activation function equiv- ariance links symmetry in parameter space to symmetry in internal representations. We then discuss how symmetry in data can induce symmetry in learned parameters, followed by a brief look at an application involving joint transformations of data and parameters. In the second part, we turn to tasks where neu- ral network parameters are treated as data, and show how parameter symmetry can be leveraged through equivariant architectures and data augmentation.
6.1 Symmetry in Parameters, Internal Representations, and Data
Symmetry parameter space is closely connocted to symmetry in internal representations, the intermediate outputs of neural networks, with both often arising from the equivariance of activation functions. In Section saw that many parameter space symmetries arise from equivariance of the activation functions
2.2
18

---

Published in Transactions on Machine Learning Research (12/2025)
0. Recall that a is equivariant to group G if it satisfies 0 0og = 9 a for g G. The set of such transformations, known as the intertwiner group, defines symmetries in both the parameter space and the internal representations of data (Godfrey et al. 2022). This property enables stitching; given two functionally equivalent networks with the same architecture but different weights, expressed as function compositions S1 11000 o f2 and S1 o o f2, an element E G can align the two via 1ogooo o f2 while preserving the output.
Symmetries in internal representations also motivate new approaches for comparing hidden activations across networks. In particular, Godfrey et al. (2022) point out that internal representations should be considered equivalent up to transformations from the intertwiner group, leading to similarity metrics that are invariant under such transformations. This perspective justifies interpreting ReLU networks through the behavior of individual neurons, whose activations are preserved under these symmetries, rather than through arbitrary linear combinations, which may obscure meaningful structure.
In two-layer RcLU networks trained with gradient descont, symmetry training data preserves symmetry in the learned parameters (Anselmi et al.] 2023). Specifically, let X C D be a dataset that is invariant. under a group G, i.e., g-X = X for all g G. This implies the loss function satisfies gX) two-layer ReLU network of the form v² o(Wx), 2), where (v, W) E θ = (At x kk and x E D Rd. Because of this data invariance, the gradient with respect to W is equivariant under G, i.e. V W) =
(00;X). Consider a
Vw LWW; X). As a result, if W is initialized to have the same symmetry as data (gWo = Wo for all g E G), this symmetry is preserved throughout training (gW, = W, for all g E G, at all time t).
While most studies examine symmetries acting on either the data space or the parameter space, it is also possible to define simultaneous transformations on both spaces that leave the loss function invariant. For example, in a one-layer lincar network f(W, b, X) -WX+L +b, the group action g (W, = Wg leaves the output unchanged. By formulating such joint group actions on data and parameters, Sonoda et al. (2025) reveal a group theoretic aspect of neural network approximation theory. Using Schur's lemma, they show that the ridgelet transform is a right inverse of the integral representation of neural networks. This means that the ridgelet transform can reconstruct a neural network that exactly represents a given data-generating function. As a result, their framework provides a constructive proof of the universal approximation theorem, rooted in the symmetry structure of the joint space.
6.2 Parameter Symmetry as Data Symmetry in Weight Space Learning
In tasks where neural network parameters are treated as data-for example, in parameter generation (Chauhan ct al. 2024) in learning representation of trained models (Schürholt et al. 2024b) -th parameter space symmetry of the input network becomes the data space symmetrr of the processing network. Designing architectures that process neural network parameters effectively has attracted increasing attention (Schürholt ict al. 2024a), partly due to the convergence of several trends: the risc of neural networks that directly encode data objects, such as implicit neural representations (Park el al. 2019); the need for weight alignment in model merging (Ainsworth et al.] 2023 Wortsman et al. (022);; and growing interest in model analysis driven by the increasing amount of publicly released trained models (Horwitz al.) (002).. Advances in methods that process neural network parameters data have enabled applications including learning
implicit neural representations (Dupont et. al. 2022). optimizing weight alignment between networks (Navon)
2024 predicting generalization ability (Unterthiner et al. 2020), recovering the dataset size used in et al.
Salama et al. 2024), and generating high-performing parameters (Abbe & Boix- Adsera 2022
Wang et al. 2024 or high-quality initializations (Knyazev et al. 2023
Respecting parameter space symmetry of the input network a desirable property of these architectures, as it ensures consistent treatment of functionally equivalent networks and improves generalization and efficiency. This concept aligns with the broader goals of geometric deep learning field that leverages structures in data to design more effective learning methods (Gerken et al. 2023 Bronstein et al.) 2021). The rest of this section reviews how these ideas are implemented through equivariant architectures and data augmentation.
One way to enforce data symmetry in processing architectures is to use equivariant neural networks (Cohen & Welling 2016 Ravanbakhsh et al., 2017 Kondor & Trivedi 2018 Maron et al., 2019)) (Figure D). Applying this idea to processing multi-layer perceptrons (MLPs), Navon et al. (2023) propose an architecture that is equivariant to permutation of the ML.P's hidden neurons. Their architecture outperforms nonequivariant
19

---

Published in Transactions on Machine Learning Research (12/2025)
(a]
(b)
Input network
Predicted properties
Input network
Predicted weight updates
meta-network
meta-network f
Test accuracy 0.8 Encoded content "3"
meta-network:
meta-network
Figure 9: Invariant and equivariant metanetworks. (a) A metanetwork f is invariant to G f(g - x) = f(x) for all input network z. Invariance is desirable in tasks such as predicting generalization performance or classifying INRs, where the output should remain unchanged under parameter symmetries. (b) A metanet-
work f is equivariant to symmetry group G' if, for all input network z, f(g -x) g- f(x). This is used in tasks such as learning to optimize (predicting update steps) or style editing for INRs (predicting weight changes in INRs that produce a given style change in the encoded content).
models on implicit neural representation classification and prediction, as well as the adaptation of classifi- cation networks lo new domains. Concurrently, Zhou et al. (2023a) develop a similar architecture that is equivariant to permutation of all neurons and imput of convolutional neural networks. Subsequently, this architecture is extended to process arbitrary weight spaces (Zhou et al. 2024) and to account for scaling symmetry in ReLU (Tran et al. 2024 although the proposed models can only learn from inputs with one fixed architecture. Lim et al. (2024a) and Kofinas et al. (024] treat neural networks as computational graphs and use graph neural networks to process the imput neural networks. Their models are thus able to learn from diverse architectures. Most recently, Kalogeropoulos et al. (2024) extend graph-based metanetworks to account for scaling symmetries arising from activation functions, achieving state-of-thh-art performance across multiple neural network processing tasks.
Building the success of equivariant architectures that leverage parameter symmetry, later work applics them to wider range of input networks and tasks. For example, Zhou et al. (023b) propose transformers that are equivariant to permutation symmetry in the imput networks, while Tran et al. (2025a) develop architectures that are equivariant to symmelry in transformers. Incorporating the knowledge of parameter symmetry into solving the weight alignment problem, Navon et al. 2024) extend the equivariant architecture in Navon et al. (2023) learn the optimal alignment between two sets of parameters. Their approach faster, produces better aligmment than traditional optimization- -base approaches, and can be used as initialization for optimization- based methods to improve their alignment quality.
Beyond equivariant networks, parameter space symmetry can also be leveraged through data augmentation. Applying symmetry transformations to existing trained parameters generates new, functionally equivalent versions of the network to be used as additional training inputs. While most existing equivariant architectures focus on permutation symmetry, Shamsiin et al. (2024) introduce scaling symmetry in ReLU networks and discrete symmetrics in sinusoidal activations generate new sets of parameters as training samples. These augmentations increase the diversity of neural represenlations for each data. object, mitigating overfitting in weight space learning and improving generalization.
7 Challenges and Future Directions
In Sections we reviewed the current understanding of parameter space symmetries, covering their theoretical foundations as well as tangible effects on the geometry loss landscapes, the dynamics of optimization, and the design of practical algorithms. In this section, we outline key rescarch directions towards a more complete theoretical foundation of parameter space symmetry, better understanding of the role of symmetry in deep learning theory, and broader applicability of symmedry-nnfommed methods.
20

---

Published in Transactions on Machine Learning Research (12/2025)
7.1 Mathematical Foundations
A fundamental challenge in understanding parameter space symmetry is to develop complete and rigorous characterization of all symmetries that preserve neural-network functions. From mathematical perspective, clarifying the existence, completeness, and structures of these symmetry groups represents foundational problem. Even seemingly simple network architectures can have nontrivial symmetry groups, and identifying them is essential for understanding loss landscapes and optimization dynamics.
Another challenge is extending the notion of symmetry from exact, punction-preservigg transformations to more flexible, data-dependent symmetries. We saw in Section 2.4 that in many neural networks with elementwise activation functions, all transformations that do not change the function are known. However, this set of symmetry is often small, as they are required to keep the loss invariant for all imput values. dependent symmetry allows for larger symmetry groups, but current analysis is limited to symmetry that preserves the loss for a single data point. Investigating the existence and structure of data-dependent symmetry for different batch sizes will to make this set of symmetry more relevant to practical scttings.
Thus far, most work has focused on layer-wise equivariances arising from adjacent activations, potentially overlooking more global invariances. Existing studies examine small components pairs of layers whose activations admit group action-but this does not prechude broader classes of symmetry that act across multiple, non-adjacent layers or repeated architectural blocks. Exploring such symmelries could substantially expand the known symmetry groups and reveal new geometric or topological structures in parameter space.
Besides pursuing a general theory, detailed investigations into architecture-specfiic symmetries are also valu- able. Models such as neural radiance fields (Mildenhall et al. 2021); which are computationally expensive to optimize, might benefit greatly from symmetry-infornn optimization mothods. Discovering and character- izing symmetries particular to these architectures would provide concrete opportunities to accelerate training and enhance scalability, thereby translating theoretical advances into immediate practical improvements.
Finally, numerical and visualization tools can help guide thcoretical characterization of parameter space symmetries. Methods for numerically constructing and visualizing functionally equivalent parameters, such as those developed by Lengyel et al. (2020), provide practical insight into the connectedness, dimensionality, and topological complexity of symmetry-indie induuced loss level sets. Numerical discovery of symmetries (Zhao et al. 2024a) and symmetry-induced structures (Martineli et al. 2024) in the parameter space may also provide intuitions on the existence and number of symmetries in given architecture. These computational approaches can be important in guiding theoretical efforts in large-scale architectures.
7.2 Deep Learning Theory
Parameter space symmetry is not merely mathematical curiosity-ii it offers lens to understand core phenomena in deep learning theory. As neural networks scale in size and complexity, symmetry provides a principled foundation for analyzing learning dynamics, the geometry of loss landscapes, and model capacity. Symmetry considerations are thus increasingly central in explaining how overparameterized networks behave, generalize, and learn in practice.
Training dynamics and implicit bias. One promising direction is to study learning dynamics through the lens of symmetry and conserved quantities. As we have seen in Scction 5 continuous symmetrics induce conserved quantities that remain fixed along gradient flows and partially parameterize the minimum. These quantities label the optimizer's position and reveal how initialization and symmetry constrain the final solution, formalizing implicit bias. However, the precise relationship between conserved quantitics, implicit bias, and generalization remains poorly understood especially in deeper architectures when SGD breaks these symmetries due to finite-step updates. Understanding how stochasticity or regularization cause the drift of these conserved quantities will be alternative route to explain phenomena in training neural networks, such as why stochastic gradient descent prefers minima that generalize.
Loss landscape geometry and generalization. Symmetry is an integral part of the geometry of the loss landscape, which is closely linked to generalization. By expanding minima into manifolds of equivalent
21

---

Published in Transactions on Machine Learning Research (12/2025)
solutions, symmetries create flat directions that confound curvature-based metrics such sharpness. Recent work by da Silva et al. (2025) addresses this issue by formulating sharpness on Riemannian quotient manifold that factors out the contimuous symmetry of transformers, recovering a strong correlation between
curvature and generalization. Developing such symmetry-aware goometric formalisms may help reconcile conflicting observations about the relationship between flatness and generalization.
Exprcssivity and approxination theory. Symmetry has direct implications for the expressivity and effective capacity of neural networks. Since many parameter configurations can represent the same function (e.g. due to permutation symmetry in multilayer perceptrons), the function class is smaller than suggested by raw parameter counts. Recent work by [Shen (022)) quantifies this, showing that factoring out symmetry leads to significantly tighter covering number bounds improving estimates by factorial factors in width. These results suggest that traditional complexity measures overestimate model capacity by ignoring symmetric rodundancies. Extending such analyses to broader symmetry groups (beyond permutations) could yield sharper generalization bounds by trealing symmetry-related parameters as equivalent in the function space.
Introducing or removing paramcter symmetrics? Architcctural design offers a mcans to either in- troduce or eliminate symmetries, which provides the opportunity to examine symmetry's effect on various aspect of deep learning. On one hand, imposing symmetry can enhance training stability and invariance. For example, Li et al. (2022) introduce a scale-invariant transformer, replacing Softmax with a normalized ReLU to create a model trainable by plain SGD yet competitive with Adam-trained BERT variants. On the other hand, removing symmetry reduces degeneracies, creates more connected minima, and sometimes makes optimization easier. Lim et al.] (2024b) propose architectures that break permutation symmetry be- tween neurons, enabling lincar mode connectivity and improving Bayesian training efficiency. Additionally, ZZyynn et al. 2025) show that removing symmetries via small perturbations can prevent capacily collapse and improve both optimization and generalization by encouraging exploration of more expressive model con- figurations. An important open question is determining which symmetries to preserve and which to break, in order to best align model behavior with the goals of a given learning task.
Taken together, these directions position symmetry not only as a descriptive tool for neural networks, but as a foundational principle for understanding learning dynamics, designing architectures, and guiding future theory in deep learning.
7.3 Applications
Optimization on loss level sets. One practical use of symmetry is to facilitate movement along loss level sets in the parameter space. Points on a fiber of the realization map can have very different neigh- borhoods (Grigsby el al. 2025). In other words, points on the same level set can be different. Early work illustrates the usefulness of this property in optimization (Armenta et al. (023). Despite initial promise, these methods have not been wide adapted, possibly due to a lack of largo-scale benchmark study and an easy to use implementation, as well the mathematical background required to adapt these algorithms to new architectures. New implementation techniques (Mishkin et al. 2023 Wu et al. 2025) and learning the teleportation destination (Zhao et al. 2024b Zamir et al. 2025) might help reduce the cost of teleportation in larger scale applications.
Besides scaling up teleportation, another promising direction is to explore level sets via symmetry transfor- mation in diverse domains. When some parts of the minima are better than others, one can use symmetry to explore the minima to find the better models. One example is continual learning, where one optimize in the minimum manifold (Farajtabar et al. 2020) and could be method for fine-tuning pre-trained models. Another example is model alignment, where symmetry is used to move models closer in the parameter space to improve model fusion (Ainsworth et al. 2023 Zhang et al.] 2025). Recent work also demonstrates that parameter space symmetries can be leveraged to align sparse subnetworks across different random initial- izations, enabling effective sparse training from scratch (Adnan et al.] 2025). Morcover, some minima. may produce lower error during quantization than others (Meleer et al. 2019 Nagel et al., [2019. Laird et al. 2025). Given known symmetry and point on minimum, we can search for these better points orbit.
22

---

Published in Transactions on Machine Learning Research (12/2025)
Reducing search space in sampling. Symmetry can be exploited beyond standard gradient-based train- ing, for instance in Bayesian methods and sampling-based optimization (Wiese et al. 2023). The parameter space explored by sampling algorithms (such as MCMC for neural network weights) may also contain redun- dancies coming from symmetry, though this aspect remains less studiod. Factoring out symmetries in such scenarios could reduce the effective search space and improve sampling efficiency. For example, if parameters related by a continuous symmetry produce the same likelihood, one could constrain the sampler to move only in the reduced space of unique configurations, thereby eliminating needless exploration of equivalent states. This idea also be used diagnostic: known symmetries provide consistency check for the quality of sampling (the sampler should visit all members of a symmetry orbit with equal probability). Dis- crete symmetry has been explored in depth (Wiese et al. 2023 Laurent et al. 2024 Xiao et al.) 2023). but lactoring out continuous symmetry might. be more appealing, since it reduces the dimension, instead of just volume, of the posterior space.
8 Conclusion
Symmetry is prevalent in neural network parameter spaces and appears in many areas of machine learn- ing, though often overlooked. Recognizing and formalizing these symmetries connects deep learning to well-established mathematical tools from group theory and geometry. Symmetry's intrinsic connection to structure improves our understanding of how neural networks work and hints at better architectures, faster optimization techniques, and new approaches to solving problems with real-world impact.
Symmetry is, of course, not the only path forward in the study of neural networks. Like modeling training dynamics using classical mechanics designing new neural networks with inspirations from neuroscience, it is example of approaching problems in machine learning from the view of pre-existing subject, allowing us to bring existing tool and insight to bear on the problem. What makes mathematical approach especially appealing that neural networks are fully artificial, abstract objects, unconstrained by the specific laws of physics or biology. By examining parameter space symmelry, we gain access to powerful mathematical frameworks that help us better understand these systems and ultimately design more effective models.
Acknowledgements
We thank Kathlén Kohn and Hancheng Min for helpful comments. This work was supported in part by the U.S. Army Research Office under Army-ECASE award W991NN-07-R-0000--00 the U.S. Department Of Energy, Office of Science, IARPA HAYSTAC Program, and NSF Grants #2205093, #2146343, #2134274, #2442658, #2134178, CDC-RFA-FT-22-0009, DARPA AIE FoundSci and DARPA YFA.
References
Emmanuel Abbe and Enric Boix-Adscra. On the non-universality of deep learning: quantifying the cost of symmetry. Advances in Neural Information Processing Systems, 2022.
Mohammed Adnan, Rohan Jain, Ekansh Sharma, Rahul Krishnan, and Yani loannou. Sparse training from random initialization: Aligning lottery ticket masks using weight symmetry. In Forty-second International Conference on Machine Learning, 2025.
Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re- re-basin: Merging models modulo permutation symmetries. International Conference on Learning Representations, 2023.
Francesca Albertini and Eduardo D Sontag, For neural networks, function determines form. Neural networks, 6(7):975-990, 1993.
Francesca Albertini, Paolo Dai Pra, et al. Recurrent neural networks: Identification and other system theoretic properties. Neural Network Systems Techniques and Applications, 3:1-41, 1995.
Alexander Altland and Ben D Simons. Condensed maller field theory. Cambridge university press, 2010.
23

---

Published in Transactions on Machine Learning Research (12/2025)
Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural computation, 10(2):251-276, 1998.
Fabio Anselmi, Luca Manzoni, Alberto D'onofrio, Alex Rodriguez, Giulio Caravagna, Luca Bortolussi, and Francesca Cairoli. Data symmetries and learning in fully connected neural networks. IEEE Access, 2023.
Marco Armenta, Thierry Judge, Nathan Painchaud, Youssef Skandarani, Carl Lemaire, Gabriel Gibeau Sanchez, Philippe Spino, and Pierro-Mare Jodoin. Neural teleportation. Mathematies, 11(2): 480, 2023.
Sanjeov Arora, Nadav Cohen, Noah Golowich, and Wei Hu. A convergence analysis of gradient descent for deep linear neural networks. International Conference on Learning Representations, 2018a.
Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization. In International Conference Machine Learning, pp. 244 253. PMLR, 2018b.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. arXiv:1607.06450, 2016.
Layer normalization.
arXiv preprint
Vijay Badrinarayanan, Bamdev Mishra, and Roberto Cipolla. Symmetry-invariant optimization in deep networks. arXiv preprint orXto:1511.01755 2015.
Bubacarr Bah, Holger Rauhut, Urrich Terstiege, and Michael Westdickenberg. Learning deep linear neural
networks: Riemannian gradient flows and convergence to global minimizers. Information and Inference: Journal of the IMA, 11(1):307 353, 2022.
Robert Bamler and Stephan Mandt. Improving optimization for models with continuous symmetry breaking. In International Conference on Machine Learning, pp. 423-432. PMLR, 2018.
Joshua Bassey, Lijun Qian, and Xianfang Li. A survey of complex-valued neural networks. arXiv preprinl arXiv:2101.19249 2021.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias viriinari trade-off. Proceedings of the National Academy of Sciences, 116(32):15849- 15854, 2019.
Gregory Benton, Wesley Maddox, Sanae Lotfi, and Andrew Gordon Wilson. Loss surface simplexes for mode connecting volumes and fast ensemblinn. In International Conference on Machine Learning, pp. 769-779. PMLR, 2021.
Frederik Benzing, Simon Schug, Robert Meier, Johannes Von Oswald, Yassir Akram, Nicolas Zucchet, Lau- rence Aitchison, and Angelika Steger. Random initialisations performing above chance and how to find them. 14th Annual Workshop on Optimization for Machine Learning (OPT2022), 2022.
Joachim Bona- Pelliselis,,,r, François Bachoc, and François Malgouyres. Parameter identifiability of a deep feedforward relu neural network. Machine Learning, pp. 1 1-63, 2023.
Johanni Brea, Berfin Simsek, Bernd Illing, and Wulfram Gerstner. Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape. arXiv preprint arXiv:1907.02911 2019.
John Bridle. Training stochastic model recognition algorithms as networks can lead to maximum mutual information estimation of parameters. Advances in neurol information processing systems, 2, 1989.
Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint 2XXi::2144.13788 2021.
David Broomhead and David Lowe. Radial basis functions, multi-variable functional interpolation and adap- tive networks. ROYAL SIGNALS AND RADAR ESTABLISHMENT MALVERN (UNITED KINGDOM), RSRF-MEMO-1148, 03 1988.
24

---

Published in Transactions on Machine Learning Research (12/2025)
Phuong Bui Thi Mai and Christoph Lampert. Functional vs. parametric equivalence of relu networks. In 8th International Conference on Learving Represenlations, 2020.
Vinod Kumar Chauhan, Jiandong Zhou, Ping Lu, Soheila Molaei, and David A Clifton. A brief review of hypernetworks in deep learning. Artificial Intelligence Review, 57(9),250, 2024.
An Mei Chen, Haw-minn Lu, and Robert Hecht-Niclsen. On the geometry of feedforward neural network error surfaces. Neural computation, 5(6):910 927, 1993.
Taco Cohen and Max Welling. Group equivariant convolutional networks. International Conference on Machine Learning, pp. 2990-2999. PMLR, 2016.
Yaim Cooper. The loss landscape of overparameterizee neural networks. arXiv preprint Xrxi 1804. 10200, 2018.
Marvin F da Silva, Felix Dangel, and Sageev Oore. Hide & seek: Transformer symmetries obscure sharpness & riemannian geometry finds it. In Forty-second International Conference on Machine Learning, 2025.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International Conference on Machine Learning, pp. 1309-1318. PMLR, 2018.
Simon S Du, Wei Hu, and Jason D Lee. Algorithmic regularization in learning deep homogencous models: Layers are automatically balanced. Newral Information Processing Systems, 2018.
Emilien Dupont, Hyunjik Kim, SM Ali Eslami, Danilo Jimenez Rezende, and Dan Rosenbaum. From data funeta: Your data point is function and you treat it like one. In International Conference Machine Learning, pp. 5694-5725. PMLR, 2022.
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in lincar mode connectivity of neural networks. International Conference on Learning Represendations, 2022.
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Ti. Orthogonal gradient descent for continual learning. In International Conference on Artificial Intelligence and Statistics, pp. 3762 3773. PMLR, 2020.
Matthew Parrugia-Roberts. Functional equivalence and path connectivity of reducible hyperbolic tangent networks. Advances in neurol information processing systems, 2023.
Charles Fefferman and Scott Markel. Recovering a food-forward not from its output. Advances in neural information processing syslems, 6, 1993.
Bella Finkel, Jose Israel Rodriguez, Chenxi Wu, and Thomas Yahl. Activation degree thresholds and expres- siveness of polynomial neural networks. Algebraic Statistics, 16(2):113- 1130 2025.
Tordan Ganev, Twan van Laarhoven, and Robin Walters. Universal approximation and model compression for radial neural networks. arXiv preprint arXiv:2177.025500 2022.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in newral information processing systems, 31, 2018.
Jan E Gerken, Jimmy Aronsson, Oscar Carlsson, Hampus Linander, Fredrik Ohlsson, Christoffer Petersson, and Daniel Persson. Geometric deep learning and equivariant neural networks. Artificial Intelligence Review, pp. 1-58, 2023.
Grzegorz Gluch and Rüdiger Urbanke. Noether: The more things change, the more stay the same. arXiv preprint 2xXt::2044.5500 2021.
Charles Godfrey, Davis Brown, Tegan Emerson, and Henry Kvinge. On the symmetries of deep learning models and their internal representations. Advances in Neural Information Processing Systems, 2022.
25

---

Published in Transactions on Machine Learning Research (12/2025)
J Elisenda Grigsby, Kathryn Lindsey, and David Rolnick. Hidden symmetries of relu networks. In Interna- tional Conference on Machine Learning. PMLR, 2023.
Elisenda Grigsby, Kathryn Lindsey, Robert Meyerhoff, and Chenxi Wu. Functional dimension of feedfor- ward relu neural networks. Advances in Mathematics, 482:110636, 2025.
Robert Hecht-Niclsen. On the algebraic structure of feedforward network weight spaces. In Advanced Neural Computers, pp. 129-135, 135. Elsevier, 1990.
Nathan W Henry, Giovanni Luca Marchetti, and Kathlén Kohn. Geometry of lightning self-attention: Identifiability and dimension. In The Thirteenth International Conference Learning Representations, 2025.
Eliahu Horwitz, Nitzan Kurer, Jonathan Kahana, Liel Amar, and Yedid Hoshen. We should chart atlas of all the world's models. arXiv preprint Xrxi::2503, 10638, 2025.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Lei Huang, Xianglong Liu, Bo Lang, Adams Yu, Yongliang Wang, and Bo Li. Orthogonal weight normal- ization: Solution optimization over multiple dependent stiefel manifolds in deep neural networks. In Proceedings of the AAAT Conference Artificial Intelligence, volume 32, 2018.
Lei Huang, Xianglong Liu, Jie Qin, Fan Zhu, Li Liu, and Ling Shao. Projection based weight normalization: Efficient method for optimization on oblique manifold in dnns. Pallera Recognition, 105:107317, 2020.
Dongsung Huh. furvature-corrected learning dynamics in deep neural networks. In International Conference PMLR, 2020.
on Machine Learning, pp.
Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. International Conference on Machine Learning, pp. 448 456. pmlr, 2015.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Aver- aging weights leads wider optima and better generalization. Conference on Uncertointy in Artificial Intelligence, 2018.
Ziwei Ji and Matus Telgarsky. Gradient descent aligns the layers of deep linear networks. In International Conference on Learning Representations, 2019.
Ioannis Kalogeropoulos, Giorgos Bouritsas, and Yannis Panagakis. Scale equivariant graph metanetworks. Advances in neural information processing systems, 17:106800-008840, 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom Brown, Benjamin Chess, Rewon Child, Scott Gray, Alee Radford, Jeffrey Wu, and Dario Amodci. Scaling laws for neural language models. arXiv preprint arXiv:2001.0888 2020.
Joe Kileel, Matthew Trager, and Joan Bruna. On the expressive power of deep polynomial neural networks. Advances in neural information processing systems, 32, 2019.
Masahiro Kimura. On unique representations of certain dynamical systems produced by contimous-time recurrent neural networks. Neural computation, 14(12):2981-22 1996,, 2002.
Boris Knyazev, Doha Hwang, and Simon Lacoste-Julien. Can we scale transformers to predict parameters diverse imagenet models? In Proceedings of the 40th International Conference on Machine Learning, volume 202, pp. 17243-17259. PMLR, 2023.
Masaki Kobayashi. Exceptional reducibility of complex-valued neural networks. IEEE Transactions on Neural Networks, 21 (7):1006001000 1072, 2010.
26

---

Published in Transactions on Machine Learning Research (12/2025)
Miltiadis Kofinas, Boris Knyazev, Yan Zhang, Yunlu Chen, Gertjan J Burghouts, Efstratios Gavves, Cees GM Snoek, and David W Zhang. Graph neural networks for learning equivariant representations of neural networks. In The Twelfth International Conference on Learning Representations, 2024.
Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in noural networks to the action of compact groups. In International Conference on Machine Learning, pp. 2747 2755. PMLR, 2018.
Yiwen Kou, Zixiang Chen, and Quanquan Gu. Implicit bias of gradient descent for two-layer relu and leaky relu networks nearly-orthogonal data. Advances in Neural Information Processing Systems, 36, 2023.
Agustimus Kristiadi, Felix Dangel, and Philipp Hennig, The geometry of neural nets' parameter spaces under reparametrization. Adoonces in Newral Information. Processing Systems, 2023.
Daniel Kunin, Javier Sagastuy-Brens Surya Ganguli, Daniel LK Yamins, and Hidenori Tanaka. Neural me- chanics: Symmetry and broken conservation laws in deep learning dynamics. In International Conference on Learning Representations, 2021.
Véra Kürková and Paul C Kainen. Functionally equivalent feedforward neural networks. Neural Compulation, 6(3):543-553 1994.
Véra Kūrková and Roman Neruda. Uniqueness of functional representations by gaussian basis function net- works. In 944: Proceedings of the International Conference on Artificial Neural Networks Sorrento, Italy, 26-29 May 1994 Volume 1, Parts 1 and 2 4, pp. 471-474. Springer, 1994.
Lucas Laird, Bo Zhao, Rose Yu, and Robin Walters. Data-free translormer quantization using parameter- space symmetry. Workshop on High-dimensional Learning Dynamics (HiLD), 2025.
Olivier Laurent, Emanuel Aldea, and Gianni Franchi. A symmetry-aware exploration of bayesian neural network posteriors. In The Twelfth International Conference Learning Representations, 2024.
Thien Le and Stefanie Jegelka. Training invariances and the low-rank phenomenon: beyond linear networks. International Conference on Learning Represenlations, 2022.
Jonathan Leake and Nisheeth K. Vishnoi. Optimization and sampling under continuous symmetry: Examples and lie theory. arXie preprint arXiv:2109.0100 2021.
Daniel Lengvel, Janith Petangoda, Isak Falk, Kate Highnam, Michalis Lazarou, Arinbjörn Kolbeinsson, Marc Peter Deisenroth, and Nicholas R Jennings. Genni: Visualising the geometry of equivalences for neural network identifiability. arXie preprint orX:x:2011.07000 2020.
Zhiyuan Li, Srinadh Bhojanapalli, Manzil Zaheer, Sashank Reddi, and Sanjiv Kumar. Robust training of neural networks using scale invariant architectures. In International Conference on Machine Learning, pp. 12656-12684. PMLR, 2022.
Derek Lim, Haggai Maron, Marc Law, Jonathan Lorraine, and James Lucas. Graph metanetworks for pro- cessing diverse neural architectures. In The Twelfth International Conference on Learning Representations, 2024a.
Derek Lim, Moe Putterman, Robin Walters, Haggai Maron, and Stefanie Jegelka. The empirical impact of neural parameter symmetries, or lack thereof. arXiv preprint ar 4rv 202311 2024b.
Mohammad M Maheri, Hamed Haddadi, and Alex Davidson. Telesparse: Practical privacy-preserving veri- fication of deep neural networks. The Privacy Enhancing Technologies Symposium (PETS), 2025.
Giovanni Luca Marchetti, Vahid Shahverdi, Stefano Mereta, Matthew Trager, and Kathlén Kohn. Algebra unveils deep learning-an an invitation to neuroalgebraic geometry. In International Conference on Machine Learning, 2025.
27

---

Published in Transactions on Machine Learning Research (12/2025)
Sibylle Marcotte, Rémi Gribonval, and Gabriel Peyré. Abide by the law and follow the flow: Conservation laws for gradient flows. Advances in Neural Information Processing Systems, 2023.
Sibylle Marcotte, Rémi Gribonval, and Gabriel Peyré. Transformative or conservative? conservation laws
for resnets and transformers. In Forty-second International Conference on Machine Learning, 2025.
Haggai Maron, Heli Ben-Hamu, Nadav Shamir, and Yaron Lipman. Invariant and equivariant graph networks. International Conference on Leurning Representations, 2019.
Flavio Martinelli, Berfin Simsek, Wulfram Gerstner, and Johanni Brea. Expand-and-clsseer: Parameter recovery of neural networks. In International Conference on Machine Learning, pp. 34895-34919, PMLR, 2024.
Eldad Meller, Alexander Finkelstein, Uri Almog, and Mark Grobman. Same, same but different: Recovering neural network quantization error through weight factorization. In International Conference on Machine Learnäng, pp. 4486-4495. PMLR, 2019.
Qi Meng, Shuxin Zheng, Huishuai Zhang, Wei Chen, Zhi-Ming Ma, and Tie-Yan Liu. G-SGD: Optimiz- ing relu neural networks in its positively scalc-invariant space. International Conference Learning Representations, 2019.
Umberto Michelucci. On the high symmetry of neural network functions. arxiv preprint or arXiv:2211.0660 2022.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99 106, 2021.
Hancheng Min, Salma Tarmoun, René Vidal, and Enrique Mallada. On the explicit role of initialization on the convergence and implicit bias of overparametrized linear networks. In International Conference on Machine Learning. PMLR, 2021.
Hancheng Min, René Vidal, and Enrique Mallada. On the convergence of gradient flow multi-layer linear models. In International Conference on Machine Learning, pp. -4880-24887. PMLR, 2023.
Aaron Mishkin, Alberto Bietti, and Robert M. Gower. Level set teleportation: the good, the bad, and the ugly. In OPT2023: 15th Annual Workshop on Optimization for Machine Learning, 2023.
Nimrah Mustafa, Aleksandar Bojchevski, and Rebekka Burkholz. Are gats out of balance? Advances in Newral Information Processing Systems, 2023.
Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling, Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1334,1334, 2019.
Aviv Navon, Aviv Shamsian, Idan Achituve, Ethan Fetaya Gal Chechik, and Haggai Maron. Equivariant architectures for learning in deep weight spaces. International Conference on Machine Learning, 2023.
Aviv Navon, Aviv Shamsian, Ethan Fetaya, Gal Chechik, Nadav Dym, and Haggai Maron. Equivariant deep weight space alignment. International Conference on Machine Learning, pp. 37376-37395. PMLR, 2024.
Behnam Neyshabur, Russ R Salakhutdinov, and Nati Srebro. Path-SGD: Path-normalized optimization in deep neural networks. In Advances in Neural Information Processing Systems, 2015.
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. The role of over-paramettriztion in generalization of neural networks. In International Conference Learning Rep- resentations, 2019.
Tohru Nitta. The uniqueness theorem for complex-value valued neural networks and the redundancy of the pa- rameters. Systems and Computers in Japan, 34(14):54 62, 2003.
28

---

Published in Transactions on Machine Learning Research (12/2025)
Emmy Noether. Invariante variationsprobleme. Nachrichten von der Gesellschaft der Wissenschaften Göllingen, haaieemmatiicc---hysaasiiee Klasse, pp. 235 255-257, 1918.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf:
Tearning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVP conference computer vision and pattern recognition, pp. 163-174, 2019.
Henning Petzka, Martin Trimmel, and Cristian Sminchisescu. Notes on the symmetries of 2-layer relu- networks. In Proceedings of the Northern Lights Deep Learning Workshop, volume 1, pp. 6-6, 2020.
Fabrizio Pittorino, Antonio Ferraro, Gabriele Perugini, Christoph Feinauer, Carlo Baldassi, and Riccardo Zecchina. Deep nelworks on toroids: Removing symmelries reveals the structure of flat regions in the landscape geometry. In Proceedings of the 39th International Conference on Machine Learning, pp. 17759 17781, 2022.
Theo Putterman, Derek Lim, Yoav Gelberg, Stefanic Jegelka, and Haggai Maron. Learning on lo- ras: Gl-equivariant processing of low-rank weight spaces for large finetuned models. arXiv preprint arXiv:2410.04400 2024.
Zhi- Hong Ran and Bao-Gang Hu. Parameter identifiability in statistical machine learning: review. Neural Computation, 29(5):1151-1203 2017.
Siamak Ravanbakhsh, Jeff Schneider, and Barnabas Poczos. Equivariance through aarameterssharing International Conference on Machine Learning, pp. 2892-2901, PMTR, 2017.
David Rolnick and Konrad Kording. Reverse-engineering deep relu networks. In International Conference on Machine Learning, pp. 8178 8187. PMLR, 2020.
Mohammad Salama, Jonathan Kahana, Eliahu Horwitz, and Yedid Hoshen. Dataset size recovery from lora weights. arXiv preprint arXiv:2406.1900 19895, 2024.
Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Advances in neural information processing systems, 29, 2016.
Lawrence Saul. Weight-balancing fixes and flows for deep learning. Transactions on Machine Learning Research, 2023.
Konstantin Schürholt, Giorgos Bouritsas, Eliahu Horwitz, Derek Lim, Yoav Gelberg, Bo Zhao, Allan Zhou, Damian Borth, and Stefanie Jegelka. Neural network weights as new data modality. In ICLR 2025 Workshop Proposals, 2024a.
Konstantin Schürholt, Michael W Mahoney, and Damian Borth. Towards scalable and versatile weight space lcarning. In International Conference on Machine Learning, pp. 43947-43966. PMLR, 2024b.
Vahid Shahverdi, Giovanni Luca Marchetti, and Kathlén Kohn. Learning on a razor's edge: the singularity bias of polynomial neural networks. arXiv preprinl vrXic:2505.118/4 2025.
Aviv Shamsian, Aviv Navon, David W Zhang, Yan Zhang, Ethan Fotaya, Gal Chochik, and Haggai Maron. Improved generalization of weight space networks via augmentations. International Conference on Machine Learning, pp. 44378 44393. PMIR, 2024.
Guohao Shen. Complexity of foed-forward neural networks from the perspective of functional equivalence. International Conference on Machine Learning. PMLR, 2024.
Berfin Şimşek, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, and Johanni Brea. Geometry of the loss landscape in overparameterizee neural networks: Symmetries and invariances. In International Conference on Machine Learning, pp. 9722-9732. PMLR, 2021.
Yang Song, Jiaming Song, and Stefano Ermon. Accelerating natural gradient with higher-order invariance. In International Conference on Machine Learning, pp. 4713 14722. PMLR, 2018.
29

---

Published in Transactions on Machine Learning Research (12/2025)
Sho Sonoda, Yuka Hashimoto, Isao Ishikawa, and Masahiro Ikeda. Unified universality theorem for deep and shallow joint-rroup-equivariann machines. International Conference on Machine Learning, 2025.
Hikari Sorensen. Understanding and collapsing symmetries in neural network parameter spaces. Bachelor's thesis, Harvard College, 2020.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. Journal of Machine Learning Research, 19(70):1-57, 2018.
Gustav Sourek, Filip Zelezny, and Ondrej Kuzelka. Lossless compression of structured convolutional models via lifting. In International Conference on Learning Representations, 2021.
Pierre Stock, Benjamin Graham, Rémi Gribonval, and Hervé Jégou. Equi- nommaliomaiiionionn of neural networks. International Conference on Learning Representations, 2019.
Héctor J Sussmann. Uniqueness of the weights for minimal feedforward nets with a given input-output map. Newral networks, 5(4):589-593 593,, 1992.
Hidenori Tanaka and Daniel Kunin. Noether's learning dynamics: Role of symmetry breaking in neural networks. Advances in Newral Information Processing Systems, 34, 2021.
Salma Tarmoun, Guilherme Franca, Benjamin D Haeffele, and Rene Vidal. Understanding the dynamics of gradient flow in overparameterized linear models. In International Conference on Machine Learning, pp. 10153-10161. PMLR, 2021.
Hoang Tran, Thicu Vo, Tho Huu, Tan Nguyen, et al. Monomial matrix group equivariant neural functional networks. Advances in Neural Information Processing Systems, 37:48628 48665,, 2024.
Hoang V. Tran, Thieu Vo, An Nguyen The, Tho Tran Huu, Minh-Khoi Nguyen-Nhat, Thanh Tran, Duy- Tung Pham, and Tan Minh Nguyen. Equivariant neural functional networks for transformers. In The Thirteenth International Conference on Learning Representations, 2025a.
Viet-Hoang Tran, Van Hoan Trinh, Khanh Vinh Bui, and Tan M Nguyen. On linear mode comnectvity of mixture-of-experts architectures. arXiv preprint 11X 2025b.
Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and Ilya Tolstikhin. Predicting neural network accuracy from weights. arXiv preprint 0rX1w:2002.11448, 2020.
Twan Van Laarhoven. L2 regularization versus batch and weight normalization. Advances in Neural Infor- mation Processing Systems, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Kai Wang, Dongwen Tang, Boya Zeng, Yida Yin, Zhaopan Xu, Yukun Zhou, Zelin Zang, Trevor Darrell, Zhuang Liu, and Yang You. Neural network diffusion. arXiv preprint arXiv:2402.1811 2024.
Yuqing Wang, Minshuo Chen, Tuo Zhao, and Molei Tao. Large learning rate tames homogeneity: Conver- gence and balancing effect. International Conference on Learning Representations, 2022.
Maurice Weiler and Gabriele Cesa. General e (2)-qquivariant steerable cnns. Advances nearul information processing systems, 32, 2019.
Maurice Weiler, Fred A Hamprecht, and Martin Storath. Learning steerable filters for rotation oquivariant cnns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 849-858 2018.
30

---

Published in Transactions on Machine Learning Research (12/2025)
Jonas Gregor Wiese, Lisa Wimmer, Theodore Papamarkou, Bernd Bischl, Stephan Günnemann, and David Rügamer. Towards efficient meme sampling in bayesian neural networks by exploiting symmetry. Tn Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD);
Research Track, pp. 459-474, 2023.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roclofs, Raphacl Gontijo-Lopes, Ari S Mor- cos, Hongseok Namkooug, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Interna- tional Conference on Machine Learning, pp. 23965- 23998. PMLR, 2022.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference computer vision (ECCV), pp. 3 19, 2018.
Zihao Wu, Juncheng Dong, Ahmed Aloui, and Vahid Tarokh. Teleportation with null space gradient projec- tion for optimization acceleration. Xiv preprint arXie:2502. 11362, 2025.
Tim Xiao, Weiyang Liu, and Robert Bamler. A compact representation for bayesian neural networks by removing permutation symmetry. UniReps: Unifying Representations in Neural Models (NeurIPS 2023 Workshop), 2023.
Ziqing Xu, Hancheng Min, Salma Tarmoun, Enrique Mallada, and René Vidal. Linear convergence of gradient descent for finite width over-parametrized linear networks with general initialization. In International Conference on Artificial Intelligence and Statistics, pp. 2262-226228022222222222222222-22 2284. PMLR, 2023.
Mianzimei Yang, Zhipeng Zhou, Jin Zhang, Yuanhao Pu, Hong Xie, and Defu Lian. Conffict-buffering oplimization by symmelry leleportation for deep long-tailed recognition. In Proceediings of the 33rd A ACM International Conference on Multimedia, pp. 1366-1375) 2025.
Mingyang Yi. Accelerating training of batch normalization: A manifold perspective. In Uncertainly in Artificial Intelligence, pp. 1128- 1138. PMLR, 2022.
Guy Zamir, Aryan Dokania, Bo Zhao, and Rose Yu. Improving learning to optimize using parameter symmetrics. arXiv preprint 2rXiv:2504.15399 2025.
Binchi Zhang, Zaiyi Zheng, Zhengzhang Chen, and Jundong Li. Beyond the permutation symmetry of transformers: The role of rotation lor model fusion. International Conference on Muchine Learning, 2025.
Bo Zhao, Nima Dehmamy, Robin Walters, and Rose Yu. Symmetry teleportation for accolerated optimiza- tion. Advances in newral information processing systems, 35:16679-16666 -6690,, 2022.
Bo Zhao, lordan Ganev, Robin Walters, Rose Yu, and Nima Dehmany. Symmetries, flat minima, and the conserved quantities of gradient flow. International Conference on Learning Representations, 2023.
Bo Zhao, Nima Dehmany, Robin Walters, and Rose Yu. Finding symmetry in neural network parameter spaces. In UniReps: 2nd Edition of the Workshop on Unifying Representations in Neural Models, 2024a.
Bo Zhao, Robert M Gower, Robin Walters, and Rose Yu. Improving convergence and generalization using parameter symmetrics. International Conference on Learning Represcntations, 2024b.
Bo Zhao, Nima Dehmamy, Robin Walters, and Rose Yu. Understanding mode connectivity via parameter space symmetry. In International Conference on Machine Learning. PMLR, 2025.
Allan Zhou, Kaien Yang, Kaylee Burns, Yiding Jiang, Samuel Sokota, J Zico Kolter, and Chelsea Finn. Permutation equivariant neural functionals. Advances in Neural Information Processing Systems, 2023a.
Allan Zhou, Kaien Yang, Yiding Jiang, Kaylee Burns, Winnie Xu, Samuel Sokota, J Zico Kolter, and Chelsea Finn. Neural functional transformers. Advances in Neural Information Processing Systems, 2023b.
Allan Zhou, Chelsea Finn, and James Harrison. Universal neural functionals. Advances in neural information processing systems, 37:104754-104777 2024.
31

---

Published in Transactions on Machine Learning Research (12/2025)
Zhipeng Zhou, Ziqiao Meng, Pengcheng Wu, Peilin Zhao, and Chunyan Miao. Continual optimization with symmetry teleportation for multi-task learning. In The Thirhy-ninth Anaual Conference. on Neural Information Processing Systems, 2025.
Zhenxun Zhuang, Mingrui Liu, Ashok Cutkosky, and Francesco Orabona. Understanding adamw through proximal methods and scale-frecncss. Transactions Machine Learning Research, 2022.
Liu Ziyin. Symmetry induces structure and constraint of learning. In Forty-first International Conference on Machine Learning. 2024.
Liu Ziyin, Hongchao Li, and Masahito Ueda. Law of balance and stationary distribution of stochastic gradient descent. arxic preprint orXiv:2308. N6W:2208.06661,, 2023a.
Liu Ziyin, Ekdeep Singh Lubana, Masahito Ucda, and Hidenori Tanaka. What shapes the loss landscape of self-supervised learning? International Conference on Learning Represenlations, 2023b.
Liu Ziyin, Yizhou Xu, and Isaac L. Chuang- Remove symmetries to control model expressivily. In The Thirteenth International Conference on Learning Representations, 2025.
32


Reviews:
[{"review_id": "Ju6jW7GAAW", "reviewer": "Reviewer_iTx9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is well-written and easy to follow.\n\n2. The paper covers diverse analysis on parameter space symmetry, including theory (group theory, geometry, and topology), optimization, conserved quantities, and interactions with data and representation symmetries, thus providing a clear understanding of parameter space symmetry.\n\nWeaknesses:\n\n1. Some expressions in the paper lack depth. For example, using “Symmetry is prevalent in deep learning architectures” as the first sentence of Section 2 looks too generic. It should benefit from a more substantial introduction.\n\n2. Some figures in the paper do not look professional. For example, in Figure 8(a), the placement and font of the legend or notation are not professional.\n\n3. The opening future research questions proposed in Section 7 are not creative and novel. They appear more like direct extensions or specific descriptions of the perspectives already covered in Sections 3-6.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This survey paper focuses on providing a overview of parameter space symmetry. It summarizes existing theoretical and algorithmic applications, emphasizing the connections to learning theory, optimization, and model analysis. In addition, the pape discusses how parameter space symmetry interacts with data and representation symmetries, offering a unified view of their role in shaping network behavior and performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper would be better to compare with the asymmetry property in some neural networks, providing a comprehensive perspective of he position of symmetry in neural networks. Perhaps involving a subsection to discuss would be better.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YMP4qq4Nln", "reviewer": "Reviewer_uQ9x", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The paper is well written, with clear structure and illustrations. The examples (e.g., Figure 2) help the readers to better understand the abstract concept.\n2. The paper is comprehensive in scope, which covers the parameter space symmetry in relation to loss, optimization and learning algorithms, which makes the survey comprehensive.\n\nWeakness:\n1. As a survey paper, the original theoretical or empirical contributions is limited. Section 2-6 primarily review the existing work, but do not consistently emphasize or synthesize new, previously unreported connections between these works.\n2. The paper is highly theoretical with minimal experimental evidence, which may limit its accessibility to a broader audience and reduce its practical impact. Even small illustrative experiments or synthetic case studies could strengthen the presentation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript thoroughly surveys parameter space symmetry by (i) reviewing definitions and examples of symmetries in neural network parameter space; (ii) analyzing the applications of parameter space symmetry; and (iii) providing related open questions and research directions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. As noted in Weakness 1, the authors should consider adding analysis in Sections 2–6 when reviewing existing work. For instance, Section 4.2 currently reads as a plain description of methods; a higher-level comparison would make this section more valuable.\n2. As noted in Weakness 2, the authors are encouraged to include small illustrative experiments. Even minimal empirical evidence could greatly help readers connect the theoretical discussions to practical outcomes.\n3. Regarding Figure 4, the color scheme could be improved for clarity. Specifically, the reviewer found the blue $GL_{d_v}$ symmetry confusing—does it correspond to the matrix multiplication shown on the right? The color coding appears inconsistent, and clearer labeling or adjustments would help avoid ambiguity.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FIjUWJ2no0", "reviewer": "Reviewer_yUTA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is **carefully written**. The Authors clearly put a lot of work in reviewing existing literature, providing a neat presentation of the results, including well explanatory tables and figures. While the paper itself does not seem to bring any original contribution (neither theoretical or empirical) a work like this one can be useful for readers interested in the topic.\n\nRegarding the weaknesses, I do think the paper gets (at times) a bit **verbose**, **rethorical**, and **speculative**, especially in the later sections when the authors switch from providing examples and definitions to elaborate more qualitatively on the roles of symmetry for the loss landscape, optimization, ... (starting  from Sections 3). Here are few examples\n\n- End of page 10: not sure I am following the logic. Two modes can be connected even if there is no element in the symmetry group that maps one in the other.\n\n- Section 3.3: model compression is, in general, done very differently than via the underlying symmetries of neural networks. While the authors mention 2 cases where symmetries are indeed considered, it is probably good to mention that this is not the current workhorse behind model compression.\n\n- End of page 11: The Authors write \"This geometric reduction _simplifies_ the landscape\". This is a bit of a vague statement, and similar comments hold for later parts (e.g. \"_easier_ to solve\" at the end of the third paragraph at page 12. Also \"more convex\" can be better phrased: maybe add \"strongly\"?).\n\n- Section 4.1: this is a very positive presentation of approaches that speed up optimization algorithms. However, to the best of my knowledge, these algorithms are not commonly used to train models at scale. Could the authors elaborate on why? Is the problem the different architectures or are these methods not giving any advantage in larger models (same question holds for Section 4.2.1)? Furthermore, the last paragraph in this section seems a bit sloppy to me: how is a \"relaxed symmetry\" defined? Maybe the Authors could spend just a few more words here.\n\n- Section 5.2: is this Section truly necessary? It seems quite redundant with 5.1 and the part on optimization.\n\n- Section 7: I would say also very verbose and with some repetitions / vague statement (e.g., \"providing a concrete handle on implicit bias\" in 4th paragraph page 21, \"could help explore flatter or higher performing regions of the loss landscape, or improve convergence by starting in a favorable orbit\" in the paragraph after, repetition on the teleportation algorithm in the first paragraph of Section 7.3)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work is a **survey** (as there are **no original results**) on some results on parameter simmetries in neural networks. Just to give some concrete examples, the Authors go through the standard permutation symmetry (Example 2.8) and positive rescaling symmetry in homogeneous networks (Example 2.3) and other actions that leave the output of a neural network unchanged. The authors later extend the notion of symmetry in weaker settings (see Figure 5, with e.g. data / distribution symmetries defined). Later, the authors connect symmetries to the existence of connections between different minima of the loss landscape and compression, and to ideas to improve gradient based optimization, namely with techniques such as gradient teleportation and parameter regularization. Finally the authors mentions existing results on conserved quantities during gradient optimization.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would kindly ask the Authors if they could consider my comments, and do a pass to remove repetition, maybe compress the presentation, and remove the most vague statements. Repeating myself: to the best of my knowledge the optimization methods that non-trivially rely on the symmetry of the parameter space are not currently used in the most modern architectures. While this does not undermine the value of this submission, I do believe elaborating on this could make the presentation more factual, and therefore better.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)