Symmetries, Flat Minima, and the Conserved Quantities of Gradient Flow
Authors: Bo Zhao
Iordan Ganev
Robin Walters
Rose Yu
Nima Dehmamy
Venue: ICLR 2023 poster (2023)
Keywords: symmetry
gradient flow
conserved quantity
flat minima
Lie group
Lie algebra

Abstract: Empirical studies of the loss landscape of deep networks have revealed that many local minima are connected through low-loss valleys. Yet, little is known about the theoretical origin of such valleys. We present a general framework for finding continuous symmetries in the parameter space, which carve out low-loss valleys. Our framework uses equivariances of the activation functions and can be applied to different layer architectures. To generalize this framework to nonlinear neural networks, we introduce a novel set of nonlinear, data-dependent symmetries. These symmetries can transform a trained model such that it performs similarly on new samples, which allows ensemble building that improves robustness under certain adversarial attacks. We then show that conserved quantities associated with linear symmetries can be used to define coordinates along low-loss valleys. The conserved quantities help reveal that using common initialization methods, gradient flow only explores a small part of the global minimum. By relating conserved quantities to convergence rate and sharpness of the minimum, we provide insights on how initialization impacts convergence and generalizability.


Published as a conference paper at ICLR 2023
SYMMETRIES, FLAT MINIMA AND THE CONSERVED QUANTITIES OF GRADIENT FLOW
Bo Zhao*
Iordan Ganev
University of California, San Diego
Radboud University iganevêcs ru.nl
bozhaolucsd. edu
Robin Walters
Rose Yu
Northeastern University
University of California, San Diego roseyulucsd. edu
r www.ateersnoothhesteeeee edu
Nima Dehmamy
IBM Research
ma dehmamy& bm. com
ABSTRACT
Empirical studies of the loss landscape of deep networks have revealed that many local minima are connected through low-loss valleys. Yet, little is known about the theoretical origin of such valleys. We present general framework for finding continuous symmetries in the parameter space, which carve out low-loss valleys. Our framework uses equivariances of the activation functions and can be applied different layer architectures. To generalize this framework nonlinear neu- ral networks, we introduce a novel set of nonlinear, data-dependent symmetries. These symmetries transform trained model such that it performs similarly on new samples, which allows ensemble building that improves robustness un- der certain adversarial attacks. We then show that conserved quantities associated with linear symmetries can be used to define coordinates along low-loss valleys. The conserved quantities help reveal that using common initialization methods, gradient flow only explores a small part of the global minimum. By relating con- served quantities to convergence rate and sharpness of the minimum, we provide insights how initialization impacts convergence and generalizability.
INTRODUCTION
Training deep neural networks (NNs) is highly non-convex optimization problem. The loss land- scape of NN, which is shaped by the model architecture and the dataset, is generally very rugged, with the number of local minima growing rapidly with model size (Bray Dean, 2007; Şimşek et al., 2021). Despite this complexity, recent work has revealed many interesting structures in the loss landscape. For example, NN loss landscapes often contain approximately flat directions along which the loss does not change significantly (Freeman & Bruna, 2017; Garipov al., 2018). Flat minima have been used to build ensemble or mixture models by sampling different parameter con- figurations that yield similar loss values (Garipov et al., 2018; Benton et al., 2021). However, finding such flat directions is mostly done empirically, with few theoretical results.
One source of flat directions is parameter transformations that keep the loss invariant (i.e. symme- tries). Specifically, moving in the parameter space from a minimum in the direction of a symmetry takes to another minimum. Motivated by the fact that continuous symmetries of the loss result in flat directions in local minima, we derive general class of such symmetries in this paper.
"Equal contribution.
Work done during internship IBM.

---

Published as a conference paper at ICLR 2023
Q= -
Our key insight is to focus on equivariances of the non- linear activation functions; most known continuous sym- metries can be derived using this framework. Models re- lated by exact equivalence cannot behave differently on different inputs. Hence, for ensembling robustness tasks, we need to find data-dependent symmetries. In- deed, aside from the familiar "linear symmetries" of NN, the framework of equivariance allows us to introduce novel class of symmetries which act nonlinearly on the parameters and are data-dependent. These nonlinear sym- metries cover a much larger class of continuous symme- tries than their linear counterparts, most any activation function. We provide preliminary ex- perimental evidence that ensembles using these nonlinear symmetries are more robust to adversarial attacks.
= minimum
0.0
0.4 L
0
U
they apply for al-
Figure 1: Visualization of the extended minimum a 2-layer linear network with loss C ||| x Points along the minima are related to each other by scaling symmetry U - Ug-1 and + gV. Conserved quantities, Q, associated with scaling symmetry parametrize points along the minimum.
Extended flat minima arise frequently in the loss land- scape of NNs; we show that symmetry-indee -inuuced flat min- ima can be parametrized using conserved quantities. Fur- thermore, we provide a method of deriving explicit con- served quantities (CQ) for different continuous symme- tries of NN parameter spaces. CQ had previously been derived from symmetries for one-parameter groups (Kunin al., 2021; Tanaka & Kunin, 2021). Using similar approach we derive the CQ for general continuous symmetries. This approach fails find CQ for rotational symmetries. Nevertheless, we find the conservation law resulting from the symmetry implies a cancellation of angular momenia between layers. To summarize, our contributions are:
A general framework based on equivariance for finding symmetries in NN loss landscapes.
2. A derivation of the dimensions of minima induced by symmetries.
3. A new class of nonlinear, data-dependent symmetries NN parameter spaces.
4. An expansion of prior work on deriving conserved quantities (CQ) associated with sym- metries, and discussion of its failure for rotation symmetries.
5. A cancellation of angular momenta result for between layers for rotation symmetries.
6. A parameterization of commetry--nduccad flat minima via the associated CQ.
This paper is organized as follows. First, we review existing literature on flat minima, continuous define continuous sym-
symmetries of parameter space, and conserved quantities. In Section metries and flat minima, and show how linear symmetries lead to extended minima. We illustrate constructions through examples of linear symmetries of NN parameter spaces. In Section 4, we define nonlinear, data-dependent symmetries. In Section 5, we use infinitesimal symmetries to derive conserved quantities for parameter space symmetries. extending the results in Kunin et al. (2021) to larger groups and more activation functions. Additionally, we show how CQ can be used define coordinates along flat minima. We close with experiments involving nonlinear symmetries, conserved quantities and a discussion of potential use cases.
RELATED WORK
Continuous symmetry in parameter space. Overparametrization in neural networks leads to sym- metries in the parameter space (Gluch & Urbanke, 2021). Continuous symmetry has been identified in fully-connected línear networks (Tarmoun et al., 2021), homogeneous neural networks (Badri- narayanan et al., 2015; Du al., 2018), radial neural networks (Ganev et al., 2022), and softmax and batchnorm functions (Kunin et al., 2021). We provide a unified framework that generalizes previous findings, and identify nonlinear group actions that have not been studied before.
Conserved quantities. The imbalance between layers in linear or homogeneous networks is known to be invariant during gradient flow and related convergence rate (Saxe al., 2014; Du et al.,
2

---

Published as a conference paper at ICLR 2023
2018: Arora et al., 2018a;b; Tarmoun et al., 2021; Min et al., 2021). Huh (2020) discovered sim- ilar conservation laws in natural gradient descents. Kunin et al. (2021) develop a more general approach for finding conserved quantities for certain one-parameter symmetry groups. Tanaka & Kunin (2021) relate continuous symmetries to dynamics of conserved quantities using an approach similar to Noether's theorem (Noether, 1918). We develop a procedure that determines conserved quantities from infinitesimal symmetries, which is closely related to Noether's theorem.
Topology of minimum. The global minimum of overparametrized neural networks are connected spaces instead of isolated points. We show that parameter space symmetries lead to extended flat minima. Previously, Cooper (2018) proved that the global minima is usually manifold with di- mension equal to the number of parameters subtracted by the number of data points. We derive the dimensionality of the symmetry-induce of infinitesimal symmetry generators and dimension of weight matrices. Şimşek et al. (2021) study permutation symmetry and show that in certain overparametrized networks, the minimum related by permutations are connected. Entezari et al. (2022) hypothesize that SGD solutions can be permuted to points on the same connected minima. Ainsworth et al. (2023) develop algorithms that find such permutations. Additional discussion on mode connectivity, sharpness of minima, and the role of symmetry in optimization can be found in Appendix A.
flat minima and show they are related to the number
CONTINUOUS SYMMETRIES IN DEEP LEARNING
In this section, we first summarize our notation for basic neural network constructions (see Appendix C for more details). Then we consider transformations on the parameter space that leave the loss invariant and demonstrate how they lead extended flat minima.
3.1 THE PARAMETER SPACE AND LOSS FUNCTION
The parameters of a neural network consist of weights' W; E Ru.xm, for each layer i, where n, and m; are the layer output and input dimensions, respectively. For feedforward networks, successive output and input dimensions match: m; 14-1- We group the widths into a tuple = (n......, f1, no)), and the parameter space becomes Param Ra X x.... R Xno, We denote an element therein as a tuple of matrices O = (W; E Re. /==- The activation of the i-th layer is a piecewise differentiable function o, : T - R.: which may or may not be pointwise. For 0 E Param and input x E Ro" the feature vector of the ith layer in feedforward network is Zi+1 (x) = W, (Z, where the juxtaposition 'Wo(Z)' denotes an arbitrary linear operation depending
the context; for example, matrix product, convolution, etc. For simplicity, we largely focus on the case of multilayer perceptrons (MLPs). We denote the final output by Fe : R - R"
defined as (2) σr (Z (z)). The "loss function" L of our model is defined as:
(1) where Data = 1R" x RL is the space of data and Cost : _ x L + R is a differentiable cost function, such as mean square eттоr cross-entropy. In the case of multiple samples, we have matrices X E "o and Y E R. whose columns are the I samples?, and retain the same notation for the feedforward function, namely, Fg R - R Most of our results concern properties of L that hold for any training data. Hence, unless specified otherwise, we take a fixed batch of data ()) C Data, and consider the loss as function of the parameters only.
Param x Data + R. L(O, (z., v)) = Cost Fx())).
Example 3.1. Two-layer network with MSE Consider a network with n = (n, h, m). the identity output activation (a (xr) = (z)),, and biases. The parameter space is Param (n) Raxh x and we denote an element as 0 = (U.V), Taking the mean square error cost function, the loss function for data (X,Y) R mm takes the form L(O, (X,)) + ())
3.2 ACTION OF CONTINUOUS GROUPS AND FLAT MINIMA
Let G be group. An action of G on the parameter space Param is a function : G x Param + Param, written as g 0, that satisfies the unit and multiplication axioms of the group, meaning id . 0 = 0 where id is the identity of G, and 91 (g2 0) = (g1.2) 0 for all Φ1, 92 E G.
For clarity, we suppress the bias vectors; all results can he extended to include bias; see appendix C.
We use capital letters for matrix data and small letters for individual samples.
3

---

Published as a conference paper at ICLR 2023
Definition 3.1 (Parameter space symmetry). The action G x Param Param is symmetry of C if it leaves the loss function invariant, that is:
Lig 0) L()),
4O Param,
g g G.
We describe examples of parameter space symmetries in the next section. Before doing so, we show how parameter space symmetry leads to flat minima (see Appendix C.6):
Proposition 3.2. Suppose G x Param - Param is symmetry of L. If 0" is critical point (resp. local minimum) of L, then so is 0 for any g E G.
The proof of this result relies on using the differential of the action of g to relate the gradient of C at 0" with the gradient at 0* We see that, if 0" is a local minimum, then so is every element of the set {g- 0" 9 E G}. This set is known the orbit of 0 under the action of G. The orbits of different parameter values may be of different dimensions. However, in many cases, there is a "generic" or most common dimension, which is the orbit dimension of any randomly chosen O.
3.3 EQUIVARIANCE OF THE ACTIVATION FUNCTION
In this section, we describe large class of linear symmetries of L using an equivariance property of the activations between layers. For accessibility, we focus on the example of two layers with output F(x) = Uo(Va for (U, V) E Param multiple layers by letting
Rm x Rhxn and x E RR. All results generalize to W; and V W 1 be weights of two successive layers deep neural network (see Appendix C.5). Let G C GLn (R) be a subgroup of the general linear group, and let : G > (R) a representation (the simplest example is x ((()) g). We consider the following action of the group G on the parameter space Param:
g U Um(g-)))) ")
g.V=gV
(3)
This action becomes a symmetry of if and only if the following identity holds:
agg=) = (=))
Vg E G, Vz E Rd
(4)
We now turn our attention to examples. To ease notation, we write GT., instead of GT,,,
Example 3.2. Linear networks A simple example of (4) is that of linear networks, where is the identity function: (xx) = x. One can take (g) = g and G GLk-
Example 3.3. Homogeneous activations Suppose the activation o A + R² is homogeneous, meaning that (1) a is applied pointwise the standard basis and (2) there exists a > such that (ccz) c² ((2) for all E Rs0 and R². Such an activation is equivariant under the positive scaling group G C GLD, consisting of diagonal matrices with positive diagonal entries. Explicitly, the group G consists of diagonal matrices g diag(c) with c (ci,..., Cn) IR
For (2) (21...., 2x) E Ra and g G, we have (gz) = Σ, ((g22) Σ, (f(() oz)) Hence, the equivariance equation is satisfied with 7(9) = g".
Example 3.4. LeakyReLU This is special case of homogeneous activation, defined as o(z) =
max(z, (z, 0) + min(z, 0), with s E R2.. We have a and T(g)= g.
Example 3.5. Radial rescaling activations A less trivial example of contimuous symmetries is the case of a radial rescaling activation (Ganev et al., 2022) where for Ra we have (2) f[][=]]a for some function f R + R. Radial rescaling activations are equivariant under rotations of the input: for any orthogonal transformation g E Ohh) (that is, g g = I) we have agg)) = gor(2) for
all E h Indeed, (gz) = [(Ilg=])) = ([[[[==])=)=
ga))), where we use the fact that ||gz|| = z gz = 2T = ||||| for g E Oh)). Hence, (4) is satisfied with ((g) = g.
We arrive at our first novel result, whose proof appears in Appendix C.6.
Theorem 3.3. The dimension of a generic orbit Param under the appropriate symmetry group is given as follows. The cases are divided based on whether h & max(n, m) or not.
Orbit Dimension
Activation Identity Homogeneous Radial rescaling
Symmetry Group GL (R) Positive rescaling O(h)
h max(n,
h 2 max(n, m) h(n + m) maa m
122
h
R
max (m0 (17.12))
4

---

Published as a conference paper at ICLR 2023
As an aside, we note that a familiar example where (4) is satisfied involves the permutation of neurons. More precisely, suppose o is pointwise and let G be the finite group of x h permutation matrices. Then (4) holds with (g)) g. However, the permutation group is finite (0-dimensional), and so does not imply the presence of flat minima.
3.4 INFINITESIMAL SYMMETRIES
Deriving conserved quantities from symmetries requires the infinitesimal versions of parameter space symmetries. Recall that any smooth action of matrix Lie group G S GL4 induces an action of the infinitesimal generators the group, i.c., elements of its Lic algebra. Concretely, let g Lie(G) G be the Lie algebra, which can be identified with certain subspace of matri- ces in gl, = Raxh, For every M E g, we have an exponential map expar R + G defined as (t) Ex= (1M)* is given by dp : g - gl, by dp(M) = jr p(expM (4)). In the case of the action appearing in (3), the corresponding infinitesimal action of the Lie algebra induced by (3) is given by:
k p: G + GT.,, is a (linear) representation, then the infinitssial action
M.U -Ud)(M,
M. MV
More generally, suppose G acts linearly on parameter space (see Appendix C for non-linear ver- sions). Set to be the dimension of the parameter space', and make the identification Param Rd by flattening matrices into column vectors. The general linear group GL ~ GLR(R) con- sists of all invertible linear transformations of Param. Suppose G is a subgroup of (Laraa) so its Lie algebra is Lie subalgebra of gla Rd For M g and 0 E Param, the infinitesimal action is given simply by matrix multiplication: M
In the case of parameter space symmetry, the invariance of L translates into the following orthogo- nality condition, where the inner product (,) Param x Param - IR is calculated by contracting all indices, e.g. (A, B) Eisk... Aigk... Bijk...
Proposition 3.4. Let G be a matrix Lie group and symmetry of L. Then the gradient vector field is point-wise orthogonal to the action of any M E g:
(VoL, . M 0) 0,
4O E Param
(6)
4 NONLINEAR DATA-DEPENDENT SYMMETRIES
For common activation functions, the equivariance (gg)) = (g) of (4) holds only for be- longing to a relatively small subgroup of GL,,, For ReLU, g must be in the positive scaling group, while for the usual sigmoid activation, the equation only holds for trivial g id. However, under certain conditions, it is possible to define a nonlinear action of the full GL, which applies to many different activations. The subtlety of such action is that is data- dependee,,,, which means that, for any g E GLa, the transformation of the parameter space depends on the input data" x.
The nonlinear action. For any nonzero vector z E IR. let (r, (r.01...... 1hh-1) be the spherical coordinates of x, and define the following h by h matrix:
zi c0s(((a5)1) (IT sin(as)
if S and I[= sin(ax) # if -+ otherwise
(Rz);5 = s
where og = 0 by convention. We observe that R2 is the product of rotation matrix and rescaling by |z|. Moreover, since 2 * 0, the first column R. is the unit vector 2/|2| and R2 has inverse given by R R Using these facts, one arrives at the following result, stated in the case of a two-layer neural network with notation from Section 3.3, and proven in Appendix
In terms of the widths, we have d = Ele T- $ 'That is, rather than being map GLn Param Param satisfying group action axioms, a data- dependent action is a map GL, x (Param x R") + Param x I2" satisfying the same axioms.
Hence, = |2] is the norm, and the i-th coordinate of is z cos(o.) [[-]] sin(ag). where on 0.
5

---

Published as a conference paper at ICLR 2023
Theorem 4.1. Suppose (2)) ) is nonzero for any 2 E RR. Then there is action GL2, (Param x I") - Param x R" given by
g (U,V,)) (URavaaava gV 2).
(7)
The evaluation of the feedforward function funciion at x unchanged: P(u,v)()) = PUUR (v ary
We emphasize that necessary and sufficient condition for the particular action of Theorem 4.1 to be well-defined is that (2) be nonzero for any 2 E IR; this is the case for usual sigmoid. Moreover, in Appendix D.2, we provide a generalization to the case where (2)) is only required to be nonzero for any nonzero 2 E IRA, a condition satisfied by hyperbolic tangent, leaky ReLU, and many other activations. The cost of such generalization is restriction to 'nonndegenerate locus' of Param x IR" where V f 0. Theorem 4.1 also generalizes to mutli-layer networks, as explained in Appendix D.3. We have the following explicit algorithm compute the action of Theorem 4.1:
0. Input: weight matrices (U, V). imput vector E IR", matrix g E GL
1. Determine the spherical coordinates of (Vv) and (gVr), and construct the matrices Ro(Va) and Ro(gVa).
2. Compute the inverse R (gV2)) Tz(gVa)| R2 (ggV))
3. Set U" U (o V and V = =gV.
4. Output: the transformed weights (U", V*). The data x E R2 remains unchanged.
Lipschitz bounds. Unlike the exact symmetries of Section 3, a data-dependent action may alter the loss in the finction space. This is evident from (7): while the transformed and original feedforward functions have the same value at x, they will differ other points. That is, if ã RR is an input value different from then F(,.)) ( * ((URnryy
(8) in general.
However, the transformed feedforward function will differ from the original one a controlled way. More precisely, when is Lipschitz continuous, we show that there is bound on how much the Lipschitz bound of the feedforward changes after the nonlinear action. The relevance of such a bound originates in the fact that we expect the distance between data points to encode important information about shared features. To be more specific, fix weight matrices (U, V), which provide the feedforward function F(ø) = UV)..... For any input vector æ E R" and matrix g GL,, the transformed weight matrices (U (o(V)) R Ro Va) .V) provide new feedforward function given by:
pla.)) (U.V)
R" RM plg.s) (U.V) (T) = Ro R 2
(8)
Proposition 4.2 (Lipschitz bounds from equivariance). Let a be Lipschitz continuous with Lipschitz constant Then F pla.z) is Lipschitz continuous with bound ||[[]]|[[|] [V]V)llll
[a(gVa))
In particular, the Lipschitz bound of the original feedforward function is M Thus, if it happens that [Vx)]llll < [x)V,,),, then the Lipschitz bound decreases when transforming the parameters. Additionally, we observe that the nonlinear action does not disrupt latent distribution of data significantly. See Appendix D.5 for proof of Proposition 4.2, which relies on iterative applica- tions of the Cauchy- Sahuhy-SShwarz inequality, as well as the fact that |R || |2|
The action described is an instance of more general framework of equiv- ariance. Specifically, map GLn Re GLn is said to be equivariance if it satisfies (1) c(id,, =) = id, for all z, and (2) e(gi, (222))(22222 2) = c(g) 82, =) for gi, 92 GLn and z. These translate directly into the unit and multiplication axioms of group", generaliz- ing N (91.2) = π (911) I (g2), and π (id,,) = id,, Every equivariance gives rise to a nonlinear action of GL., Param Rh given by g (U, V, preserving the loss and only if the following generalization of (4) holds:
General equivariance.
two conditions
(Uelg, Vx)--1gV, a.). This action a symmetry
General Equivariance:
(gz) c(g. 2))
Vgg E GL, Vz R²
explicit example of such an equivariance o(g, 2) Ro(gz) (aa) and Proposition 4.2 general-
izes to any general equivariance by replacing [=(V=)]]lg] with [[o[g Vx) -1I-
[[]]]]
'In fact, defines GT,, -qquivariaa structure on the tangent bundle of R.

---

Published as a conference paper at ICLR 2023
5 CONSERVED QUANTITIES OF GRADIENT FLOW
We have shown that continuous symmetries lead us along extended flat minima in the loss landscape. In this section, we identify quantities that (partially) parameterize these minima. We first show that certain real-valued functions on the parameter space remain constant during gradient flow. We refer to such functions conserved quantittis. Applying symmetries changes the value of the conserved quantity. Therefore, conserved quantities can be used to parameterize flat minima.
Gradient flow (GF). Recall that GD proceeds in discrete steps with the update rule 0z+1 = 0. =V)((0, where ε is the learning rate (which in general can be symmetric matrix), and t t= 0, 0,, are the time steps. In gradient flow, we can define a smooth curve in the parameter space from choice of initial values to the limiting local minimum without discretizing over time. The curve is a function of a continuous time variable / E IR, and velocity of this curve at any point is equal the gradient of the loss function, scaled by the negative of the learning rate. In other words, the dynamics of the parameters under GF are given by:
θ (t) dd(t)/ -
(10)
From an initialization O(0) at O, GF defines trajectory ((t) Param for & Ryo, which limits to a critical point. In this way, GF is a continuous version of GD.
Conserved quantities. A conserved quantity of GF a function Q Param -> IR such that the value of Q at any two time points s, t a along GF trajectory is the same: ((O(s)) (()()). In other words, we have dQ(O(t))/dt /tt = O. Note that, if f R + I is any function, and Q is conserved quantity, then the composition f Q is also conserved quantity. Several conserved quantities of GF have appeared in the literature, most notably layer imbalance Qimb = ||W]l
|W,, 2 (Du al., 2018) for each pair of successive feedforward linear layers (a(x) = z), and its full matrix version Q. = W.T. W, W.
We now propose generalization of the layer imbalance by associating a conserved quantity to any infinitesimal symmetry. As in Section 3.2, suppose a matrix Lie group G acts linearly on the parameter space. Then, from (6), we have the identity (VoC, M 0) 0 for any element M in the Lie algebra g. Using the gradient flow dynamics (10), this identity becomes:
E 0 M. M- -0
(1I)
In other words, the velocity at any point of a gradient flow curve is orthogonal to the infinitesimal action. For simplicity, we set the learning rate to the identity: & (all results generalize to symmetric E.) The following proposition (whose proof is elementary and well-known) provides a way of 'integrating' equation (1I), in the appropriate sense, in order to obtain conserved quantities:
Proposition 5.1. Suppose the action of G Param is linear? and leaves L invariant. For any M E g. there is a conserved quantity Qu Param - R given by m ()) = (0, M A).
While Proposition 5.1 directly links the infinitesimal action to conserved quantities, it has the limita- tion that the conserved quantity corresponding to anti-symmetric matrix M -M" in g is con- stantly zero, and we do not obtain meaningful conserved quantities. Instead, we can only conclude that flow curves satisfy the differential equation (11). Fixing a basis (0), ... 0²) for Param ~ Rd, this equation becomes Σiej Marrz 0 where (raj, are the polar coordinates for the point (O., 0,) E R² (see Appendix C.9.5). In summary, we find:
Meg differential equation 0T M0
symmetric M conserved quantity Q (0) 0T M0
anti-symmetric M differential equation Eies mirr2 01j 0
Conserved quantities parametrize symmetry flat directions. metry changes the values of the conserved quantities QM (Figure I). Indeed, for M E and E G,
We observe that applying sym-
For simplicity, we also assume that G is closed under taking transposes, and acts faithfully on the parameter space. These assumptions generally hold in practice; see Appendix C for version with fewer assumptions.
7

---

Published as a conference paper at ICLR 2023
have (g 0) = Q Mg for all 0 E Param, applying the group action" transforms the conserved quantity Qu to Q.T As discussed in Section 3. applying to a minimum 0 of L yields another minimum g 0"; hence applying symmetries leads to partial parameterization of flat minima. Note that, in general, we may lack sufficient number of Q to fully parameterize a flat minimum. For example, in the linear network UVx, G GL,, and flat minima generically have h?2 dimensions, whereas the number of independent nonzero QM is h 1 1)/2, which is the dimension of the space of symmetric matrices M M in gl
In gradient descent, the values of these conserved quantities may change due to the time discretiza- tion. However, the change in Q is expected be small. For example, two-layer linear networks, the change of Q is bounded by the square of learning rate. Appendix E contains derivations and empirical observations of the magnitude of change Q.
Relation to Noether's theorem. In physics, Noether's 's theorem (Noether, 1918) states that contin- uous symmetries give rise conserved quantities. Recently, Tanaka & Kunin (2021) showed that Noether's theorem can also be applied to GD by approximating it as second order GF. We show that the limit where the second order GF reduces to first order GF (10), results from Noether's theorem reduce
conservation law (M,, VC) = 0 (6). In short, using Noether's theorem, the conserved Noether current is JM e Joss with JOM Mo,e-1o). 1)).
In the limit 7 0, using (10), Jom (Mo VL) 0 and the conservation dJM /dt implies Joss 0, meaning we recover (6). Details appear in Appendix B.
Examples. We present examples of conserved quantities for two-layer neural networks. all of which directly generalize to the multi-layer case. See Appendix C.9 for full derivations (which heavily rely on properties of the trace). We adopt the notation of Section 3.3.
Example 5.1. General equivariant activation Suppose o is equivariant under a linear action of subgroup S GLi,, (R), so that (z) = (gz). Then the two-layer network F(z) = (V((V is invariant under G, as is the loss function. For symmetric M E g. Proposition 5.1 yields the following conserved quantity:
Q Param - R,
- = Tr [VT MV] - Tr (UT (
(12)
Indeed, this follows from the fact that M (U,V(UVV = (-Udd( MV), as in (5).
Example 5.2. Imbalance in linear layers Suppose the network is linear. Then (2) = 2 and the loss is invariant under GL (R). For symmetric M we have the conserved quantity QM (U, Tr [[VV* - -U"UMM] Moreover, each component of the matrix VVT - - is conserved.
Example 5.3. Homogeneous activation under scaling Suppose o is a homogeneous activation of degree a. Let G (R>x))* be the positive rescaling group, so that (gz) g² for any E G and E Ra. Note that the Lie algebra of G consists of all diagonal matrices in gl,- so that, in particular, each M E g is symmetric. Since dr (M) = aM for any M E g. we obtain the conserved quantity Q (U, V) = Tr [VVVT UU Using the basis M Est- we see that Q = diag [VVT - TU is conserved (here, diag[ AA] is the leading diagonal). Special cases of this are LeakyReLU and ReLU with a = 1.
Example 5.4. Radial rescaling activations Let a be such a radial rescaling activation. As in Section 3.3, the orthogonal group G Ohh) is symmetry of L. The Lie algebrag g = 50% comprises anti-symmetric matrices, and so Proposition 5.1 yields no non trivial conserved quantities. However, using the canonical basis of g so,, given by F[UJ = FEt- Fex (so [kl] indicates anti-symmetrized indices), one uses equation (11) to deduce the following novel result (see Appendix C.9):
Theorem 5.2. When is radial rescaling activation, we have:
VVT VVT + -ーーー-ーTー
(13)
for any V) E Param, where the dots indicate derivatives with respect gradient flow.
Expanding the (k;,) entry of the matrix on the left-hand-side of (13), we obtain:
E SU + Em / dvr s:kt = 0, where (Fu, k and (TVT, s:bts Φyz
"Note that this procedure only works if gf"" Mg helongs to g, which is the case the examples we consider.
8

---

Published as a conference paper at ICLR 2023
(5)
[b)
(c)
[d)
On 0033
Qm1.03e-24
0.6
crig
Qwil1 3
100
0.5
(ropg)
107
Frequenic
0wd2.3
pem, rewp
10-1
0.01
2ee35 Traning Steps
66+60
Eigenvalues of Messian
FCSM opailon
tranetomes
Figure 2: Overview of empirical observations with more details in Appendix G, H. and I. (a) In a two-layer neural network, the convergence rate depends on the conserved quantity Q. (b) The distribution of the eigenvalues of the Hessian the minimum is related to the value of Q. (c) The ensemble created by group actions has similar loss values when & is small. (d) The ensemble model improves robustness against fast gradient sign method attacks.
are the 2D polar coordinates of the points (Usk: Usi and (V VT). This is analogous to the "angu- lar momentum" in 2D, that is: A 2 r² Intuitively, Theorem 5.2 implics that in every 2D plane (kt, 1), the angular momenta of the rows of U and the columns of V sum to zero. These results also apply to linear networks F(x:) UVx, since rotational symmetries are linear.
APPLICATIONS
We present a set of experiments aimed at assessing the utility of the nonlinear group action and conserved quantities". summary of the results are shown Figure 2. conserved quantities can impact convergence rate and generalizability. We also find the nonlinear action to be viable for ensemble building to improve robustness under certain adversarial attacks.
We show that the value of
Exploration of the minimum. While Q is often unbounded, common initialization methods such as (Glorot & Bengio, 2010) limit the values of Q to a small range (Appendix F). As a result, only a small part of the minimum is reachable by the models. Symmetries allow us explore portions of flat minima that gradient descent rarely reaches.
Convergence rate and generalizability. Conserved quantities are by definition unchanged during gradient flows. By relating the values of conserved quantities to convergence rate and model gen- eralizability, have access properties of the trajectory and the final model before the gradient flow starts. This knowledge allows us to choose good conserved quantity values at initialization. In Appendix G, we derive the relation between Q and convergence rate for two example optimiza- tion problems, and provide numerical evidence that initializing parameters with certain conserved quantity values accelerates convergence. In Appendix H, we derive the relation between conserved quantities and sharpness of minima in a simple two-layer network, and show empirically that Q values affect the eigenvalues of the Hessian (and possibly generalizability) in larger networks.
Ensemble models. Applying the nonlinear group action allows us obtain ensemble without any retraining or searching. We show that even with stochasticity in the data, the loss is approximately unchanged under the group action. The ensemble has the potential to improve robustness under adversarial attacks (Appendix I).
DISCUSSION
In this paper, we present a general framework of cquivariance and introduce a new class of nonlinear, data-dependent symmetries of neural network parameter spaces. These symmetries give rise to conserved quantities in gradient flows, with important implications in improving optimization and robustness of neural networks. While our work sheds new light onto the link between symmetries and group, it contains several limitations, which merit further investigation. First, we have not been able to determine conserved quantities in the radial rescaling case, only a differential equation that gradient flow curves must satisfy. Second, one major contribution of this paper is the non-linear group action of Section 4. However, our formulation only gurantees full GL,, equivariance for batch size k = 1. In future work, we plan to explore more consequences and variations of this non-linear group action, with the hope of generalizing greater batch size. Finally, in many cases, parameter space symmetries lead to model compression: i.c., finding a lower-dimension space of parameters with the same expressivity of the original space.
Our code is available https: //gi htt.t/gtttt/ttt//// thub. www./oooooooeeeeooooooooo -T-Be--TTI--- aaaa//rrriiiii eent ow- Symmet ry.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was supported in part by U.S. Department Of Energy, Office of Science grant DE- SC0022255, U. S. Army Research Office grant W911NF-20-1-0832 and NSF grants #2134274 and #2146343. I. Ganev was supported by the NWO under the CORTEX project (NWA.1160) (60.18..16) R. Walters supported by the Roux Institute and the Harold Alfond Foundation and NSF grants #2107256 and #2134178.
REFERENCES
Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. International Conference Learning Representations, 2023.
Marco Armenta, Thierry Judge, Nathan Painchaud, Youssef Skandarani, Carl Lemaire, Gabriel Gibeau Sanchez, Philippe Spino, and Pierre-Marc Jodoin. Neural teleportation. Mathematics, 11(2):480, 2023.
Sanjeev Arora, Nadav Cohen, Noah Golowich, and Wei Hu. A convergence analysis of gradient descent for deep linear neural networks. International Conference on Learning Representations, 2018a.
Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization. In International Conference on Machine Learning, pp- 244-253. PMLR, 201
Vijay Badrinarayanan, Bamdev Mishra, and Roberto Cipolla. Sommetry-invariian optimization in deep networks. arXiv preprint arXiv:1511.0175: 2015.
Robert Bamler and Stephan Mandt. Improving optimization for models with continuous symmetry breaking. In International Conference on Machine Learning, pp. 423-432. PMLR, 2018.
Gregory Benton, Wesley Maddox, Sanae Lotfi, and Andrew Gordon Wilson. Loss surface simplexes for mode connecting volumes and fast ensembling. In International Conference on Machine Learning. pp. 769-779. PMLR, 2021.
Alan J Bray and David S Dean. Statistics of critical points of gaussian fields on large-dimensional spaces. Physical review letters, 98(15) 2007.
Shih-Kang Chao, Zhanyu Wang, Yue Xing, and Guang Cheng. Directional pruning of deep neural networks. Advances in Neural Information Processing Systems, 33: 13:86-113999 2020.
Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. International Conference on Learning Representations, 2017.
Yaim Cooper. The loss landscape of overparameterized neural networks.
arXiv preprint
1rXiv:1804.10200 2018
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. International Conference on Machine Learning, pp. 1019-1028. PMLR, 2017.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International conference on machine learning, pp. 1309- 1318. PMLR, 2018.
Simon Du, Wei Hu, and Jason D Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. Neural Information Processing Systems, 2018.
Omer Elkabetz and Nadav Cohen. Continuous vs. discrete optimization of deep neural networks. Advances in Neural Information Processing Systems, 34, 2021.
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. International Conference on Learning Representations, 2022.
10

---

Published as a conference paper at ICLR 2023
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness- aware minimiza- tion for efficiently improving generalization. International Conference on Learning Represen- tations, 2020.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode nectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp- 3259-3269. PMLR, 2020.
C Daniel Freeman and Joan Bruna. Topology and geometry of half-rectified network optimization. In 5th International Conference on Learning Representations, ICLR 2017, 2017.
Iordan Ganev, Twan van Laarhoven, and Robin Walters. Universal approximation and model com- pression for radial neural networks. arXiv preprint arXiv:2107.0255500 2022.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information pro- cessing systems, 31, 2018.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings. 2010.
Grzegorz Gluch and Rüdiger Urbanke. Noether: The more things change, the more stay the same. arXiv preprint arxi (itv:2/04.05500 2021.
Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong. and Richard Socher. Using mode con- nectivity for loss landscape analysis. arXiv preprint 0tti::/800.06999 2018.
Haowei He, Gao Huang, and Yang Yuan. Asymmetrie valleys: Beyond sharp and flat local minima. Advances in neural information processing systems, 32, 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation, 9(1::1442, 1997.
Dongsung Huh. Curvatune-corrected learning dynamics deep neural networks. In International Conference on Machine Learning, pp. 4552-4560. PMLR, 2020.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. Conference on Uncertainty in Artificial Intelligence, 2018.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. Inter- national Conference Learning Representations, 2017.
Minyoung Kim, Da Li, Shell X Hu, and Timothy Hospedales. Fisher sam: Information geometry and sharpness aware minimisation. In International Conference on Machine Learning, pp. 11148- 11161. PMLR, 2022.
Daniel Kunin, Javier Sagastuy- Brena, Surya Ganguli, Daniel LK Yamins, and Hidenori Tanaka. Neural mechanics: Symmetry and broken conservation laws in deep learning dynamics. In Inter- national Conference on Learning Representations, 2021.
Jonathan Leake and Nisheeth K Vishnoi. Optimization and sampling under continuous symmetry: Examples and lie theory. arXiv preprint arXiv:2109.01080, 2021.
Qi Meng, Shuxin Zheng, Huishuai Zhang, Wei Chen, Zhi-Ming Ma, and Tie-Yan Liu. G-SGD: Optimizing relu neural networks in its positively scale- sale-invariiat space. International Conference on Learning Representations, 2019.
Hancheng Min, Salma Tarmoun, René Vidal, and Enrique Mallada. On the explicit role of initializa- tion on the convergence and implicit bias of overparametrized linear networks. In International Conference on Machine Learning. PMLR, 2021.
Behnam Neyshabur, Russ R Salakhutdinov, and Nati Srebro. Path-SGD: nath-normalized optimiza- tion in deep neural networks. In Advances in Newral Information Processing Systems, 20155

---

Published as a conference paper at ICLR 2023
Emmy Noether. Invariante variationsprobleme. Nachrichten der Gesellschaft der Wis- srahhemmascch-Pyysaalaaaaaa Physikalische Klasse, pp. 235-257 1918.
senschaften zu Götrnngen, Mathematisch
Henning Peizka, Michael Kamp, Linara Adilova, Cristian Sminchisescu, and Mario Boley. Relative flatness and generalization. 35th Conference Neural Information Processing Systems, 2021.
Fabrizio Pittorino, Antonio Ferraro, Gabriele Perugini, Christoph Feinauer, Carlo Baldassi, and Riccardo Zecchina. Deep networks on toroids: Removing symmetries reveals the structure of flat regions the landscape geometry. In Proceedings of the 39th International Conference on Machine Learning, pp. 17759-17781, 2022.
Levent Sagun, Utku Evei, V Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical analysis of the hessian of over-parametrizee neural networks. arXiv preprint arXiv:/706.044454 2017.
Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dy- namics of learning deep linear neural networks. arXiv preprint arXiv:1312.61206 2014.
Berfin Şimşek, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerst- ner, and Johanni Brea. Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances. In International Conference on Machine Learning, pp. 9722-9732. PMLR, 2021.
Hidenori Tanaka and Daniel Kunin. Noether's s learning dynamics: Role of symmetry breaking in neural networks. Advances in Neural Information Processing Systems, 34, 2021.
Salma Tarmoun, Guilherme Franca, Benjamin D Haeffele, and Rene Vidal. Understanding the dynamics of gradient flow in overparameterized linear models. In International Conference on Machine Learning. pp. 10153-10161, PMLR, 2021.
Twan Van Laarhoven. L2 regularization versus batch and weight normalization. Advances in Neural Information Processing Systems, 2017.
Andre Wibisono and Ashia C Wilson. On accelerated methods in optimization. arXiv preprint 10xx::1500.03616. 2015.
Lei Wu, Zhanxing Zhu, al. Towards understanding generalization of deep learning: Perspective of loss landscapes. arXiv preprint arXiv: 10239, 2017.
Yuqian Zhang, Qing Qu, and John Wright. From symmetry to geometry: Tractable nonconvex problems. arXiv preprint arXiv:2007.06753, 2020.
Bo Zhao, Nima Dehmamy, Robin Walters, and Rose Yu. Symmetry teleportation for accelerated optimization. Advances in Neural Information Processing Systems, 2022.
Pan Zhou, Jiashi Feng, Chao Ma, Caiming Xiong, Steven Chu Hong Hoi, et al. Towards theoreti- cally understanding why sgd generalizes better than adam in deep learning. Advances in Neural Information Processing Systems, 33:21285... 21296, 2020.
12

---

Published as a conference paper at ICLR 2023
CONTENTS
Introduction
2
Related Work
3
Continuous symmetries in deep learning
3.1
The parameter space and loss function
3.2 Action of continuous groups and flat minima
3.3 Equivariance of the activation function
3.4
Infinitesimal symmetries
4
Nonlinear data-dependent symmetries
5
Conserved quantities of gradient flow
Applications
7 Discussion
Additional related works
14
B Relation to Noether's theorem
15
C Neural networks: linear group actions
15
C.1 Jacobians and differentials
16
C.2 Neural network parameter spaces
16
C.3 Action of continuous groups and infinitesimal symmetries
17
C.4 The hidden symmetry group
18
C.5 Linear symmetries
18
C.6 Linear symmetries lead to extended, flat minima
19
C.6.1 Flat directions in the two-layer case
20
C.7 Conserved quantities
22
C.8
Conserved quantities from group action
22
C.8.1 Anti-symmetric case
24
C.9
Examples of conserved quantities for neural networks
24
C.9.1
Conserved quantities for linear networks
24
C.9.2
Conserved quantities for linear networks: two-layer case
25
C.9.3
Conserved quantities for ReLU networks
25
C.9.4
Conserved quantities for ReLU networks: two-layer
26
C.9.5
Conserved angular momentum for radial rescaling networks
26
C.9.6
Conserved angular momentum for radial rescaling networks: two-layer case
27
13

---

Published as a conference paper at ICLR 2023
C.10 Jacobians: special cases
27
D Neural networks: non-linear actions group actions
28
D.I Rotations
................................
28
D.2 Non-linear action: two-layer case
................................
D.3 Non-linear action: multi-laye layer case
................................
30
D.4 Discussion: increasing the batch size
................................
D.5 Lipschtiz bounds
32
E Gradient descent and drifting conserved quantities
32
E.1 Change in Q in gradient descent (linear layers)
32
E,2 Empirical observations
Distribution of Q under Xavier initialization
33
G Conserved quantity and convergence rate
34
G.1 Example I: ellipse
G.2 Example 2: radial activation functions
36
G.3 Experiments
H Conserved quantity and generalization ability
39
H.1 Example: two-layer linear network with ID parameters
40
H.2 Experiments: two-layer networks.
41
I Ensemble models
41
A ADDITIONAL RELATED WORKS
Mode connectivity / flat regions/ / ensembles In neural networks, the optima of the loss functions are connected by curves volumes, on which the loss is almost constant (Freeman Bruna, 2017; Garipov et al., 2018; Draxler et al., 2018; Benton et al., 2021; Izmailov et al., 2018). In these works, various algorithms have been proposed
find these low-cost curves, which provides low-cost way to create an ensemble of models from single trained model. Other related works include linear mode connectivity (Frankle et al., 2020), using mode connectivity for loss landscape analysis (Gotmare et al., 2018), and studying flat region by removing symmetry (Pittorino et al., 2022).
Sharpness of minima and generalization Recent theory and empirical studies suggest that sharp minimum do generalize well (Hochreiter & Schmidhuber, 1997; Keskar et al., 2017; Petzka al., 2021). Explicitly searching for flat minimum has been shown to improve generalization bounds and model performance (Chaudhari et al., 2017: Foret et al., 2020; Kim et al., 2022). The sharpness of minimum can be defined using the largest loss value in the neighborhood of minima (Keskar et al., 2017; Foret et al., 2020; Kim et al., 2022), visualization of the change loss under perturbation with various magnitudes on weights (Izmailov et al., 2018), singularity of Hessian (Sagun et al., 2017), or the volume of the basin that contains the minimum (approximated by Radon measure (Zhou et al., 2020) product of the eigenvalues of the Hessian (Wu et al., 2017)). Under most of these metrics, however, equivalent models can be built to have minimum with different sharpness but same generalization ability (Dinh et al., 2017). Applications include explaining the good generalization of SGD by examining asymmetric minimum (He et al., 2019), and new pruning algorithms that search for minimizers close to flat regions (Chao et al., 2020).
14

---

Published as a conference paper at ICLR 2023
Parameter space symmetry and optimization While sets of parameter values related by the symmetries produce the same output, the gradients at these points are different, resulting in different learning dynamics (Kunin et al., 2021; Van Laarhoven, 2017). This insight leads to a number of new advances in optimization. Neyshabur et al. (2015) and Meng et al. (2019) propose optimization algorithms that invariant symmetry transformations on parameters. Armenta al. (2023) and Zhao et al. (2022) apply loss-invariant transformations on parameters to improve the magnitude of gradients, and consequently the convergence speed.
The structures encoded in known symmetries have also led to new optimization methods and insights of the loss landscape. Bamler & Mandt (2018) improves the convergence speed when optimizing in the direction of weakly broken symmetry. Zhang et al. (2020) discusses how symmetry helps in obtaining global minimizers for a class of nonconvex problems. The potential relevance continu- ous symmetries in optimization problems was also discussed in Leake & Vishnoi (2021), which also provides an overview of Lie groups.
B RELATION TO NOETHER'S 'S THEOREM
We will now show how the approach in Tanaka & Kunin (2021) relates conservation law dQ/dt dt (0, M0) O. Assuming small time-step 7 < 1, we can write GD as ((++T) -()) =
V (()(t)). Expanding the l.h.s to second order in r and discarding 0(73) terms defines the 2nd
order GF equation
d0
do
2nd order GF:
+
-VL.
(14)
dt
2 dt2
Here E/T. To use Noether's theorem, the dynamies (i.e. GF) must be variational (Euler- Lagrange (EL)) equation derived from an "action" S(0) (objective function), which for (14) is the time integral of Bregman Lagrangian (Wibisono Wilson, 2015) L
S(0) ( ((())) - dtel/ (0,=
c(0)
(15)
where 0 : R + Param is trajectory (flow path) in Param, parametrized by t. The variational EL equations find the paths 7n' which minimize the action, meaning /8y/
Noether's theorem states that if M E is symmetry of the action S(0) (15) (not just the loss (()),, then the Noether current JM conserved
aL
Noether current:
JM = Me.
= -/l/T (Mo, = e JoM:
dJM = e/T Jom +
1
dJow = 0, Jos = JoME t/r (16)
Conservation:
dt
dt
Tanaka & Kunin (2021) also derived the Noether current (16), but concludes that because L(0, 0) + L()). the symmetries "broken" and therefore doesn't derive conserved charges for the types of symmetries we discussed above. However, while Tanaka & Kunin (2021) focuses on 2nd order GF, we note that our conserved Q were derived for first order GF, which is found from the t - limit of 2nd oder GF. In this limit L + et/T L and thus symmetries of L also becomes symmetries of L. When r > 0, 2nd order GF reduces to e 0 = -VL the conserved charge goes to
lim Jom (Mo. ; VL) Joar(0) (0) lim -t/r O, r-+0
(17)
7-20
which means that we recover the invariance under infinitesimal action (6). In fact, for linear sym- metries and symmetric M E g. Joss dQM/ /dt
C NEURAL NETWORKS: LINEAR GROUP ACTIONS
In this appendix, provide extended discussion of the topics of Section 3, including full proofs of all results. Specifically, after some technical background material on Jacobians and differen- tials, specify conventions for neural network parameter space symmetries. In contrast the
15

---

Published as a conference paper at ICLR 2023
discussion of the main text, (I) assume that neural networks have biases, and (2) focus the multi-layer case rather than just the two-layer case. We then turn our attention to group actions the parameter space that leave the loss invariant, and the resulting infinitesimal symmetries. The groups we consider are all subgroups of a large group of change-of-basis transformations of the hidden feature spaces; call this group the 'hidden symmetry group'. We also compute the dimensions of generic extended flat minima in various relevant examples. Finally, we explore consequences of invariant group actions for conserved quantities.
C.I JACOBIANS AND DIFFERENTIALS
In this section, summarize background material on Jacobians and differentials. We adopt notation and conventions from differential geometry. Let U C R" be an open subset of Euclidean space IR", and let F U + R be a differentiable function. Let F,...., Fm U + IR be the components of F, so that F(u) (Fi()).... Fm (u)). The Jacobian of F, also know as differential F, at U is the following matrix of partial derivatives evaluated at
0F 8g
0 aF 0rg
0f
8F
0x,
Da.,
dF..
a.
a.
0xg
The differential d'F,, defines a linear map from R" to Rm that is, an element of mR Observe that if F itself is linear, then, as matrices, dF,, F for all points u E U. If G Rm - R2 is another differentiable map, then the chain rule implies that, for all E R", we have:
d(G F) = dGF(u) o dF..
In the special case the m 1, the differential is E I2 is defined as the transpose of the Jacobian dF,;:
vector, and the gradient ..F of F
(dFu)
BF
OF De
1111= = 0x, 1n
OF
E E
C.2 NEURAL NETWORK PARAMETER SPACES
Consider neural network with L layers, input dimension no = n. output dimension ny = m, and hidden dimensions given by 01,.... B1-1- For convenience, we group the dimensions into tuple (No, R1, ..,.... 12).. The parameter space is given by:
Param(n) = R xnx-1 x 18 x 122m X R X x RR x P x x R"
We write an element therein as a pair 0 (W, b) of tuples W (W,, W...... W3) and b (bi,...,h) b.). so that W; is a n, x fi-1 matrix and b; is vector in RR for i clear from context, we write simply Param for the parameter space. Fix a piecewise differ-
T. When
entiable function o; : R": + I for each i = 1,... L. The activations can be pointwise (as is conventionally the case), but are not necessarily so. The feedforward function
F= Fo R Rm
corresponding to parameters 0 = -WWbb E Param with activations a; is defined in the usual recursive way. To be explicit, we define the partial feedforward function Fø.; = F. R" + R to be the map taking E _ to oz (W;F,-1111 + bi), for i = I,... L. with Fo idg Then the feedforward function F = Fz. The "loss function" L our model defined as:
(18) where Data Re x R is the space of data (i.e., possible training data pairs), and Cost RVy x R + R is a differentiable cost function, such as mean square error or cross-entropy. Many, if not most, of our results involve properties of the loss function that hold for any training data. Hence, unless specified otherwise, we take fixed batch of training data { (x,, 9:) 3 S Data, and consider the loss to be function of the parameters only.
C Param Data
C(O, (a. v)) = Fo
16

---

Published as a conference paper at ICLR 2023
The above constructions generalize to multiple samples. Specifically, instead of R and y E RR one has matrices X E R=.*k xk and Y E R x whose columns are the k samples. Additionally, one uses the Frobenius norm of n, X k matrices to compute the loss function. The i-th partial feedforward function is Fo,: R -> R and we have the feedforward function Fo : R x + R xk (We use the same notation as in the case k= 1; the number of samples will be understood from context.)
Example C.I. Consider the case L = 2 with no biases and no output activation. The dimension vector (no,7)) n1; n2) (n, h, m), so the parameter space is Param(n. h, m) R x Rmxh. Taking the mean square error cost function, the loss function for data (X, Y) E Rank x Rm takes the form C(O, (x, = = [V)l.) where 0 (W2, Wi) (U, V) E Param.
C.3 ACTION OF CONTINUOUS GROUPS AND INFINITESIMAL SYMMETRIES
Let G be group. An action of G on the parameter space Param is function : G X Param + Param, written as g 0, that satisfies the unit and multiplication axioms of the group, meaning
T. 0 = 0 where 7 is the identity of G, and 91 (02 0) = (9192) ) A for all 91, 92 E G. Recall that we say an action x Param -> Param symmetry of Param with respect L if it leaves the loss function invariant, that is:
(19) The groups within the scope of this paper are matrix Lie groups, which are topologically closed subgroups G C GT,, (R.) of the general linear group of invertible n x n real matrices. Any smooth action of such a group induces an action of the infinitesimal generators of the group, i.c., elements of its Lie algebra. Concretely, let Lie(G) TyG be the Lie algebra, which can be thought of as a certain subspace of matrices in gl, = R'**, or (equivalently) as the tangent space 10 at the identity I of G. For every matrix M E g. have exponential map expas E - G defined as (+M)
Lgg. 0) = (()),
4A E Param, g GEG
(t) = =k=o m! vector field M:
Given an action of G on Param, the infinitesimal action of M is
d Infinitesimal action of M vector field: Mo
(exp 9).
AA Param. (20)
dt 1-00
Hence, the value of the vector field M at the parameter value A is given by the derivative at zero of the function & + (exp,r(t) - O). In the case of a parameter space symmetry, the invariance of L translates into the orthogonality condition in Proposition 3.4, where the inner product ()
Param Param - is calculated by contracting all indices, e.g. (A, Eijk... Aijk.. Bisk...
Proof of Proposition 3.4. The gradient is the transpose of the Jacobian (see Section C.I), so the & left-hand-side becomes dCo (de (t) 0)). We compute:
(d) (exp -0) dt
dLe
((exp 0) = dt la
( =
dt u
where the first equality follows by the chain rule, the second equality uses the invariance of C. and the third equality follows since (()) does not depend on t.
Next, we comment on the case of a linear action. Observe that the parameter space is a vector space of dimension d = dim (Pamaammm) = Ele n2(1 + 14-1). Hence, there is an isomorphism Param ~ Rd which flattens any tuple 0 = (W,b) b) into vector in Rd. We can identify the group GLr(aar of all invertible linear transformations of the parameter space with the group GL. of invertible d x d matrices, and the Lie algebra of (Param) with وا. Rdxd Suppose G acts linearly on the parameter space. Then we can identify" G with subgroup of GT.(Param). acting on Param with matrix multiplication. Similarly, we can identify the Lic algebra g of G with Lie subalgebra of gla = dd In this case, the infinitesimal action is given by matrix multiplication:
Me = M 0. We recover Equation (6).
HoHene, elements of the Lie algehra are 'velocities' the identity of G. More precisely, for every Lie algehra element E, there is path " (-e,e) + G whose value at 0 is the identity of G and whose derivative (i.c., velocity) at zero is &. Modding out by the kernel, if necessary.
17

---

Published as a conference paper at ICLR 2023
C.4 THE HIDDEN SYMMETRY GROUP
Consider neural network with L layers and dimensions n. The hidden symmetry group correspond- ing dimensions is defined the following product of general linear groups:
Gl,,,
GL
An element is tuple of invertible matrices g (gi,.... 9L-1), where 9: E GLm,, Consider the action the hidden symmetry group on the parameter space given by
Giyat () (aramn()
g (W, b) = (g.W. gibi)
(21)
where go idng and gr. idn,, This action amounts to changing the basis at each hidden feature space. The Lie algebra of p is
gluii gl,, x .... gla...
and the infinitesimal action of the tuple M E gla is given by:
gl tialen + Param(n)
M (WWWb) (MWW W-M-1, M,).)
where we set My and Mr to be the zero matrices.
Example C.2. In the case L with dimension vector n (n, h, m) and biases. The hidden symmetry group is GL2 with Lie algebra gh,, The action of the group and the infinitesimal action of the Lie algebra are given by:
GL ) Param(n, h, m) = Rhxx x mxc
9 (V,U) = (DV.Ug-*)
g1 ( (aram h, m) Rhhn x ma
M (V,U) = (MV,
C.5 LINEAR SYMMETRIES
Consider a feedforward mully-connected neural network with widths (no. n2), so that the
parameters space consists of tuples of weights and biases 0 = (W; E 1 b, E RR _ i=1" For
each hidden layer 0 < L, let G, be a subgroup of GL,,, and let n. G, + GL (R) be
a representation (in many cases, we take [g(g) = g). Hence the product G = G1 x G is subgroup of the hidden symmetry group GT Gugu Define an action of on Param via
Vg ........9L) E G,
g- W; g. W
g g;b;
(22)
where gn and gr. the identity matrices idn, and id., respectively. This is version of the action defined in (21), with the addition of the twists resulting from the representations n;,
We now consider the resulting infinitesimal action. For each the representation n; induces Lie algebra representation d(T;))
+ gha,- The infinitesimal action of the Lie algebra =
01 0z induced by 22 given by:
VM (M. M2)
M W, M.W W:d(==-)(MM-i) M. M,d.
(23)
The proof of the first part of the following Proposition proceeds by induction, where the key com- putation is that of from (4). The second part relies on (6).
Proposition C.I. Suppose that, for each = 1,..., L, the activation o; intertwines the two actions of G. that is, (g(22)) (e(9))0((z)) for all g: G5, 2, R., Then:
1. (Combined equivariance of activations) The action of G G1 x ... x GL defined in 22 is a symmetry of the parameter space.
2. (Infinitesimal equivariant action) The action of g = B1 X X Or defined in 23 satisfies (VoL, M. for all 0 Param and all M g.
Proof. Let g = (g:) E G, so that 9: E G, C GLn As usual, set go = id,, and gt. = id,,- Also me and n be the identity maps GI,,, and Gla respectively. Fix parameters
18

---

Published as a conference paper at ICLR 2023
and imput value RR. We show by induction that the following relation between the partial feedforward functions holds:
Fg)),(z)= TH (..
for i = O...., L. The base step is trivial. For the induction step, we use the recursive definition of the partial feedforward functions:
F = = 9:b:)
b,)
b.) - (g:) Fo,)(x))
Hence 0 and g- 0 define the same feedforward function. Since the loss function depends on the parameters only through the feedforward function, the first claim follows. The second claim is consequence of Proposition 3.4.
Note that two-layer case of the above result amounts to (4) (the argument can be simplified in that case). While Proposition C.1 is stated for feedforward networks, it can easily be adopted to more general settings, such networks with skip connections and quiver neural networks. Denoting the Jacobian of 0; IR + R at 2 E g by d(o;)2 E Rnxx,, the infinitesimal form of version of (g(g)) TH (g)o;
(d(o.),, Mz) = dr; (M
vM E gln,
(24)
When the activation is pointwise, we have Mr @ df(z) ==(=dd()))))) (M))).))) where @ denotes ele- mentwise multiplication. We now illustrate Proposition C.1 through examples in the two-layer case with input dimension n, hidden dimension h, hidden activation 0, and output dimension
Example C.3. Linear networks For linear networks, we have (xx) = x. One can take π (gg) and G GL, (R).
Example C.4. Homogeneous activations Suppose the activation a : IRAh + Rh is homogeneous, so that (1) is applied pointwise in the standard basis, and (2) there exists
such that
(c2) = cº (2) for all c E Rz0 and E TRA,, These a are equivariant under the positive scaling group G GLn, consisting of diagonal matrices with positive diagonal entries. For g E G. we have diag(c) is a diagonal matrix with с = (CI..... Cb) E Rn. For 2 = (zi,..., =n) E RR,, we have (gz) = E, aCjzz)) E, (2,) = gz) Hence, the equivariance condition holds with (g) = n Since dr dM(M) a.M for any element M of the Lie algebra g of G, the infinitesimal version of rescaling invariance of homogeneous becomes M 2 O (2(2) a.)o(())
Example C.5. LeakyReLU This a special case of homoegeneous activation, defined as o(2) = 1, and (g)) g. Since o==) 20 2z)(2),
max(z, 0) min(z, 0). with E Rso. We have infinitesimal equivariance becomes Mr @ (z()) = Mz(2).
Example C.6. Radial rescaling activations A less trivial example of continuous symmetries is the case of a radial rescaling activation (Ganev et al., 2022) where for 2 E I² \ {0). (()) = f[] for some function f R R.. Radial rescaling activations are equivariant under rotations of the input: for any orthogonal transformation g E O(h) (that is, g" = I) we have a(gz) = ga(2) (z) for all E RAA. Indeed, (gz)) = [(Ilg=1))(g)) ((f([== (2)),, where ||g=|| = z"'g² gz 2fx |=||| for E O(h). Hence, (4) is satisfied with (gg) g.
the fact that
C.6 LINEAR SYMMETRIES LEAD TO EXTENDED, FLAT MINIMA
In this section, we show that, in the case of a linear group action, applying the action of any element of the group to local minimum yields another local minimum. This fact is corollary of a more general result; in order to describe it and remove ambiguity, we include the following clarifications. Let G be a matrix Lie group acting as a linear symmetry. Fix basis (0,,...,0,) 02) of the parameter space. The gradient VgL of the loss C at a point E Param in the parameter space as another vector in Param ~ Rd, whose i-th coordinate is the partial derivative dc 29; Hence, it makes sense
to apply the group action to the gradient: V.L. We regard vectors in Param ~ Rd as column vectors with d rows. Thus, the transpose of any vector is row vector with d columns. In the case of the gradient, its transpose at f matches the Jacobian dC, E x L (see Appendix C.I), that is: d. (VoL)". Alternative notation for the Jacobian is dLo,, 02 l., where we use
19

---

Published as a conference paper at ICLR 2023
dummy variable and 0g E Param specific value. As noted above, interested in matrix Lie groups G C (L) = GL(Param). and assume that the matrix transpose g belongs to G for any g E G. These assumptions hold in all examples of interest. We have the following reformulation of Proposition 3.2:
Proposition C.2. Suppose the action of G on the parameter space is linear and leaves the loss invariant. Then the gradients of L any 0g and g Og are related as follows:
g v Vacf
Vg E G, VOg- E Paaam
(25)
0 a critical point (resp. local minimum) of then so is g 0
Sketch of proof. Let Tg Param + Param be the transformation corresponding to E G. The the Jacobian dLe, is given by:
0L dCa. 00 0.
T(CoTg)
ac
ST. 00
= dL o T
00
00
1.
where we use the definition of the Jacobian, the invariance of the loss (L Tg L), the chain rule, and the linearity of the action. The result follows from applying T g- on the right to both sides, and taking transposes (see Appendix C.1). The last statement follows from the invariance of under the
action of G, and the fact that V C = 0 at critical point 0 of C.
We conclude that, if 0 is a critical point, then the set {g 9 E G} belongs to the critical locus. This set is known as the orbit of 0 under the action of G, and is isomorphic to the quotient 0) is the stabilizer subgroup of 0 in G. In the
G/Stabe(()) where (t) case of a linear action, the orbit is a smooth manifold. While the results above imply that the critical locus is a union of G-orbits, they do not imply, in general, that the critical locus is a single G-orbit, They also do not rule out the case that the stabilizer is somewhat 'large' subgroup of G, in which case the orbit would have low dimension. However, in many cases, there a topologically dense subset of parameter values 0 Param whose orbits all have the same dimension. We call such an orbit a 'generic' orbit. We now turn our attention to examples of two-layer networks where such a generic orbit exists.
{g= E G g
C.6.1 FLAT DIRECTIONS IN THE TWO-LAYER CASE
Recall that the parameter space of two-layer network is Param Rmxh x Rhxn where the dimension vector is (m, h, n), and we write elements as (U, V). The action of GL, is g- (U, V) (Ug-1,9) .VV). Let Param" C Param be the subset of pairs (U, V) where each of U and V have full rank. This is open dense subset of Param, and is preserved by the -La-action action.
Proposition C.3. The Go.b-orbit of each element of Param has dimension
dim(Orbit) = 112 -aa h n) max(0, h m).
Proof. Fix (U, V) E P Suppose S n, so that 1:2 max (0, h - n)) max(0, h m) = h2. Then V E hix defines surjective linear map, so has right inverse V Raxh If g E GL, belongs to the stabilizer of (U, V)) then we have gV = V. Applying V on the right to both sides, obtain g = id,, Thus, the stabilizer of (U, ?) is trivial and the orbit has dimension equal the dimension of the group, namely h.2 The case S m is similar.
Now suppose h max(n, m.. In this case, 2 max(0, n) max(0, h m) h(n+ mm.
Set Un jimm 0] E Ra and Vo id,"
E RRhxn, so that the last h rows of Uo are zero,
and the last n columns of Vo are zero. Then, by the rank assumption, there exists 91 E GLn such that Ug Un, and there exists 02 GL, such that 92 'V Vo. Without loss of generality, we can take 91 and 92 such that det(gg) > 0 and det > 0. Thus, both Φi and ga belong to the component of the identity in GL
Next, consider the action of G = GL2 on full rank matrices in Rm and gaxa individually. We have that (U) gi gtatab(UU))g) and Slabo(V) 92Stabo (V))g The stabilizer in
20

---

Published as a conference paper at ICLR 2023
GLn of the pair (U, V) Param can be written as:
Stabo(U, V) [g G Ug- U and gV =V
(26)
= SU) (Va (o1Staba(Uo)o n (22Stabo(Vo)92
(27)
(28)
Since 91 and 92 belong to the connected component of the identity, the dimension of (g1 (tabb ') [g2Stabe(Vo)gg) Hence we reduce the problem to computing the dimension of Staba(Uo) (Staab(VV))
equal the dimension" 1 of Stabo n Stabb
To this end, observe that matrix g belongs to Sta (resp. Stabe (V))) and only is of the form:
idn 0
id,, (resp. g 0
*
g
*]
where the lower left E (R(mm))xm and the lower right * E GTa-m (resp. upper right E [Rnx(h-n) and lower right * E GL are arbitrary. m 2 n, taking the intersection amounts to considering matrices of the form:
idn 0
0 0
g =
0
where the rows and columns are divided according to the partition h + (m n2) (h m). If m 2 n, taking the intersection amounts to considering matrices of the form:
Fidm 0
0
id - m
0
0
where the rows and columns are divided according to h = m + (m-m))+- m)+ (h-n). In both cases, the
dimension of the intersection is (h r2)(h m) 122 hn hm + nm. We obtain the dimension of the orbit as: 122 (h ( m) = h(n + m) - nm..
Recall that the symmetry group for homogenous activations is the coordinate- wise positive rescaling subgroup of GL,, consisting of diagonal matrices with positive entries along the diagonal. We denote this subgroup as T. (h). Similarly, the symmetry group for radial rescaling activation is the orthogonal group (h)). For linear networks, the activation is the identity function, so the symmetry group is all of GL
Corollary C.4. The orbit of a point in Param" under the appropriate symmetry group is given by:
Type of activation
Symmetry group
Dimension of generic orbit
Linear
GLn
h2
max(0, n) max(0, h m)
Homogeneous
(+)
min(h, max(n,m))
if S max(n, otherwise
Radial rescaling
O(h)
hh- (h-max(m,n)
Proof. Adopt the notation of the proof of the above Proposition. The stabilizer in (h) (Uo, Vo) is the intersection of the stabilizer in GLn, of (Uo, Vo) and (h). This intersection is casily to have dimension max(0, h max(n, m)). Subtracting this from dim(T, (h)) h, we obtain the result for the homogeneous case. For the orthogonal case, the stabilizer in O(h) of (Un, Vo) is
the intersection of the stabilizer in GT. of (U,, Vo) and O(h). This intersection has dimension
if h < max(n, m) and (h- (A-maxn,m)) otherwise. Subtracting from dim(O(h)) (2), we obtain the result for the radial rescaling case.
Explicitly fix continuous path The dimension of (F)taaa(((())) (n)) (vv()))taao(V)))2(() ") is constant along this path.
|0. 1] such that (0) is the identity in G and (1) gt. for i 1,2.
21

---

Published as a conference paper at ICLR 2023
C.7 CONSERVED QUANTITIES
We now turn our attention to gradient flow and conserved quantities. In this section, we give formal definition of conserved quantity. Let V R be the standard vector space of dimension d. Suppose : V + R is differentiable function. Let Flow, : V + V be the flow for time t along the reverse gradient vector field, so that:
[1 + Flowr(v)) = v.
di
Note that Flowo the identity on V, and, for any 8, the composition of Flow, and Flow, is Flows++ s+t- We will write v(t) for Flow,(") (v), that d(s) C.) A conserved quantity is function Q V + that satisfies either of the following equivalent conditions:
For any t, we have Q(v(t)) = Q(v).
2. Let Q(e) # lo ((v(t)) be the derivative of Q along the flow. Then Q = 0.
3. The gradients of Q and L are point-wise orthogonal, that is, Qv.Q, V. for all E V.
The equivalence of (1) and (2) are immediate. To show the equivalence of the third and second statements, let and compute:
(V. VoL) dQu(o) V,C. = dQuso) d d dt v(1) (Q o()))
di
where we use the definition of the flow the second equality, and the chain rule in the third.
We note that, if f : P + I is any function, and Q is conserved quantity, the f Q is also served quantity. Additionally, any linear combination of conserved quantities is again a conserved quantity. Let Conserv(V, L) denote the vector space of conserved quantities for the gradient flow of L: IR. For any u E V, there a map:
V. Conserv()) (V, L.) - T.V = Rd.
Q~Q
taking a conserved quantity to the value of its gradient at v. By the above discussion, the map V, is valued in the kernel of the differential dh,,
C.8 CONSERVED QUANTITIES FROM A GROUP ACTION
Let G be subgroup the general linear group GLa Thus, there is linear action of G on V = Rd. Suppose the function L is invariant for the action of G, that is,
L(g v) L(v)
V Vg G.
Let g Lie(G) be the Lie algebra of G, which is a Lie subalgebra of gla = BRdd, The infinitesimal action of g on V is given by g V V, taking (M, v) to Mu.
Proposition C.5. Let L R be a G-invarian function, and let M E
1. For any E V, the gradient of T, and the infinitesimal action of M are orthogonal:
(V.L. = 0.
2. Suppose (a,b) b) + V is a gradient flow curve for L. Then:
(G(1))² Mn(t) = Vt (a,
3. Suppose M belongs to g. Then the function
Qm QM:V+R
v 4 of Mu
is conserved quantity for the gradient flow of T.
22

---

where inc: G Rdxd is the inclusion (which passes through the inclusion of G in (R)((R) ev, Rdxd + Rd be the evaluation map at v, and the left vertical map is the constant map at {u}. Indeed, the clockwise composition is g + Lgvv). which is equal the constant map at g L(v). The chain rule implies that taking Jacobians at the identity 1 of G' results in the commutative diagram on the right. where g is the Lie algebra of G, identified with the tangent space of at the identity; the tangent space of the vector space IR at is canonically identified with IR and the the zero appears because the tangent space of single point is zero. The derivative of the inclusion map is Published as a conference paper at ICLR 2023
Proof. For the first claim, observe that the invariance of L implies that the left diagram commutes:
inc
Rdxd
eVe
Rd
dinci
Raxd
d(ov=hia
R
L
dbc
(0)
R
IR
L
the inclusion > gla Rddd, while the derivative the evaluation map is itself as it is linear map. Hence, for M E g. we have:
0 dL,, a d(evv)la, dinci (M) dL, ev. (Mt) (V).
The first claim follows. The second claim is consequence of the first claim, together with the defini- tion of a gradient flow curve. For the third claim, we take the derivative of the composition of Q M with gradient flow curve y:
d dt
= (Qaro7)= (4()))(MM + (²()
(M(1,, +M)
=-(Vxc) (M + (M))(())
= =-(Vrr04,, - (V (V) (²(((t))
Both terms in the last expression are constantly zero by the second claim. Hence Qai is constant on any gradient flow curve, and so it is conserved quantity.
We summarize some of the results and constructions of this section diagrammatically. Let g a Lie subalgebra in general). Observe
denote the vector space of symmetric matrices in g (this that g n g" is the set of all M such that M E g. Let Infing (V, L) denote the vector space of infinitesimal conserved quantities for the gradient flow of the G-invariant function L V R. We have:
g
M-QQ
(V, -> Conserv(V, L.)
2y
where the map g gg"" - g grym takes M to its symmetric part 1/ (M + .M). while the map yn g E gf is the natural inclusion. We note that g n gf is the Lie algebra of the group G n GT while grym is in general not Lie algebra. By definition, the vector space Infing L) is the image of the map M + QM defined on g g" is straightforward to verify the following result:
Corollary C.6. The map M Qu establishes isomorphism of vector spaces: gym Infing (V, L).
As discussed in Section 3.2, applying a symmetry g to minimum 0' of L yields another minimum 0" Using flattened E IRd it easy to show that acting with g changes some Qu (0) = 0" Me. < 1 be g G close to identity. We have Qm 0) = QM + T (M M MM' 0 + O Thus, whenever T M + MM 0, applying changes the value Q. Therefore, QM can be used to parameterize the flat minima. However, for anti-symmetric M, could not find nonzero Q explicitly.
Let g (t) = + tM', with M' g and 0 n
23

---

Published as a conference paper at ICLR 2023
C.8.1 ANTI-SYMMETRIC CASE
Suppose M E g is anti-symmetric, so M -M y (a, - R be a gradient flow curve.
Write in coordinates as 7 (m.....7a Proposition C.5 implies that (+(())* My = 0. Hence we have:
0
Σ i
mij (3r7g whs))= Σ
mish
11 7/
Σ
marri
where rij = (() is equal to v /x(1)² - (1) and 0ú- = (1g(1) is the angle between the i-th coordinate axis and the ray from the origin to the projection of 7(1) to the pilane One verifies the last equality using the definition of 0,g in terms of the arctangent of the quotient ///n. We see that (ray(t), (t)) are the polar coordinates for the point (7a(t), (t)) E R
0 a for some nonzero a E R, and so: -a 0
Case d = 2. Then M
o My a (t) -n (r)))
where r(() and (() are polar coordinates. Setting the final expression equal to zero, we obtain that ((1) constant along any flow line 7()) that begins away from the origin.
0
b7
Case d = 3. Then M
=
a
0
c
for some a, b, c E IR, and so:
b
C
0
12t My = ari2012 + bri + cr2g023
C.9 EXAMPLES OF CONSERVED QUANTITIES FOR NEURAL NETWORKS
We now compute conserved quantities for gradient flow on neural network parameter spaces in the case of linear, homogeneous, and radial networks. In each case, state results first in the for general multi-layer network, and then for the running example of two-layer network. Throughout, @ denotes the Hadamard product of matrices, defined by entrywise multiplication. We also set TMM) to be the sum of all entries in a matrix M. We note that, for square matrices M and N of the same size, T(M N) Tr(M" N), which is the same as the inner product of the flattened versions of M and N.
The notation for the running example of two-layer network is as follows. We set the input and output dimensions both equal to one, hidden dimension equal to two, and no bias vectors. The hidden layer activation is a R R22. The parameter space Param = 22x1 x R1x2 with
elements written as pair of matrices: (U,V)
[w1
U2]
11
The hidden symmetry group
U2
is GL2, with action given by:
GR) x Param + Param,
(g. U. V) + (U.V) (Ug-i.9V).
The Lie algebra gl, of GL2(R) (R) consists of all two-by-two matrices.
C.9.1 CONSERVED QUANTITIES FOR LINEAR NETWORKS
Suppose a neural network with L layers has the identity activation a; = id,, in each layer, so that the resulting network is linear. Then it is straightforward to verify that the networks with parameters g. (W, b) and (W, b) have the same feedforward function. Consequently, the loss is invariant for the group action: its value the original and transformed parameters is the same for any choice of training data. (As we will see below, for more sophisticated activations, one needs to restrict to subgroup of the hidden symmetry group achieve such invariance.)
24

---

Published as a conference paper at ICLR 2023
Suppose M gla is such that M, gh,, is symmetric for each i. The conserved quantity implied by Proposition C.5 is:
Qm (r(W, @ M.W) + r(b, o M,b,) T(W++ @ Wc)
Tr ((WW) + b,, W W.+1) M.)
We examine these conserved quantities in the following convenient basis for the space of symmetric matrices in gisin For j = 1,..., and {k, {} S [ , Nigh ,igh set:
ing) kk
if
(k.e] =
(nn) (E + (())
#
where (x(1)) ke is the elementary n, n2 matrix with the entry in the k-th row and é-th column equal to one, and all other entries equal to zero. Then one computes:
Ry-1
ni+1
Q FU
( b) = 00 1000
Σ
(1+1) (+1) w of
In other words, we take the sum of the following three terms: the product of the k-th and (-th entries of the bias vector by. the dot product of the k-th and f-th rows of Wj, and the dot product of the k-th and (-th columns of W In particular, we see that every entry of the matrix
μ. (W, b) == W.W? b, W W+11 E gla,
a conserved quantity valued in gh,, rather than in IR Additionally, we have moment map:
Q Param
gl Mook:
(W, - M (u,(W,b) Mc)
C.9.2 CONSERVED QUANTITIES FOR LINEAR NETWORKS: TWO-LAYER CASE
In the two layer case of linear network, have that that the single hidden activation is the identity: o = idy RR2 + R The hidden symmetry group is GL2 with Lie algebra all of gl2- The space of symmetric matrices in gاو is spanned by the matrices:
0
E11
0
E22
0
1
and
E(,2)
1 0
The corresponding conserved quantities are:
(U,(U,)) V) of 2f (U,,(U 03 2i
QEa.., (U, V) = 0102 U1
Thus, obtain three-dimensional space of conserved quantities. (Since GT.2 also contains the orthogonal group (22), Equation 29 below holds along any gradient flow curve.)
C.9.3 CONSERVED QUANTITIES FOR RELU NETWORKS
The pointwise ReLU activation commutes with positive rescaling, so we consider the subgroup of the hidden symmetry group consisting of tuples of diagonal matrices with positive diagonal entries, that
G ig E GLy
9: Diag(s1......... s1...., Sn,) 8y 0}
This subgroup, also known as the positive coordinate- rescaling subgroup, is isomorphic to the product (Rx0))[=]] n, Its Lie algebra spanned by the elements ElJ) defined above, for j --1 and k ...... ny. The conserved quantity implied by Proposition C.5 is:
Q D (. b)
(62)
2
W
2
Σ
(w ((++))))))
In other words, we take the sum the following three terms: the square of the k-th entry of the bias vector by. the norm of the k-th row of W,, and the norm of the k-th column of W,+1-
25

---

Published as a conference paper at ICLR 2023
C.9.4 CONSERVED QUANTITIES FOR RELU NETWORKS: TWO-LAYER CASE
In the two-layer case, the positive rescaling group is:
G = {
0 E
2L2(R)
I1 and 92 are positive,
9a
The Lie algebra of G is the two-dimensional space of diagonal matrices in gl2 (with not necessarily positive diagonal entries). other words, is spanned by the matrices E11
1 0
and E22 =
0
0
One computes the conserved quantities corresponding to these elements as:
QE. (U, = v Qe (U, = 12 2
Hence there is a two-dimensional space of conserved quantities coming from the infinitesimal action.
C.9.5 CONSERVED ANGULAR MOMENTUM FOR RADIAL RESCALING NETWORKS
Suppose each o; is radial rescaling activation (z) = Ae([z))z, where A R + R is the rescal- ing factor. Each such activation commutes with orthogonal transformations, so we consider the subgroup of the hidden symmetry group consisting of tuples of orthogonal matrices:
G {g E GT gigt id..; for all
The Lie algebra of this subgroup consists only of anti-symmetric matrices, and there are no infmitesiiall-ctiion conserved quantities. However, given an anti-symmetric matrix M; E g for each i, any gradient flow curve satisfies the following differential equation (encoding conservation of angular momentum):
C.ii.WW + r(b. @ M,b.) - r(W41 @ WH41M)) =
(ef. Section C.8.1). An equivalent way to write this equation is:
1-1
Σ Tr (WWW: + babj + W W Wi+1) M.) 1=1
Indeed, one uses the facts that 7(A 2 Tr B), Tr(AT) for any two matrices A, B of the appropriate size in each case. Using a basis of anti-symmetric matrices, one can show that the matrix
Tr(A), and Tr(AB) Tr(BA),
Di ((Wb)) = W,W? W.W. + b. i, WT. W WTIWW+1 E gt.,
is equal to zero: i (W, = O. Note that v; depends on taking derivatives with respect to the flow. In fact, v4 is more properly formulated as function on the tangent bundle T(Param) of Param, which is then evaluated on the gradient flow vector field. Similarly, we have a moment map 7 ((Paam)) + giynee, and the gradient flow vector field contained in the preimage of zero. We omit the details.
basis for the space of anti-symmetric matrices in gl given by:
E(1) = (n(s) F(ny)
where =1 L 1, and k (1,..., nz} satisfy is < The differential equation correspond- ing to E is given by:
2 T b +
Σ
Wyyks,t es
2
O &W
+ Σ
2 wo & workkr
where (Nbyske, ee) are the polar coordinates of the image of by under the projection R - R2 which selects only the k-th and ε-th coordinates. Similarly, for any pair matrix entries have a projection g Xn, + R² and can take the polar coordinates of the image of W, under this projection.
26

---

Published as a conference paper at ICLR 2023
C.9.6 CONSERVED ANGULAR MOMENTUM FOR RADIAL RESCALING NETWORKS: TWO-LAYER CASE
In the two-layer radial rescaling case, suppose the dimension vector is (n, h, m). and that there no bias vectors. For U E Rmxh, V E Raxn and M E so(h), we have:
M M0M.O). 0) ((U,)) =
- -T(OO" TTV(MMV)
-Tr(VVTM) -TMOTTT) M) =Ir(VVTM) -T(UTÜM) =Tr (AMM))TT(((() [T[(VVT ((VVT+UTÚ) M
Hence obtain the differential equation:
Tr [(VT++U*U) +UTT M
In the case where (n, h, m) (1, 2, 1), we have the two by two orthogonal group:
G = 0(2) = {g E 2 و" 00'giid = id}
The Lie algebra G consists of anti-symmetric matrices in gl2, and contains no non-zero symmetric matrices. Hence, we do not obtain any conserved quantities from the infinitesimal action in this case.
However, using the element 0
1
E g. we obtain that the following differential equation holds
0
along any gradient flow curve:
02 + rr.dv =
(29)
where (ru: fy) are the polar coordinates of (u1, u2) E IR2 and similarly for (rv, fy). Note that the left-hand side of Equation 29 is function of t; so if (a, b) - is gradient flow curve, then more precise version of the equation is (r) 7) (t) (Bu o 7) (f) (rē o7) (t) (0v 7) (t) 0 for I.
C.10 JACOBIANS: SPECIAL CASES
We conclude this appendix with a side remark on special cases of the Jacobian formalism.
Manifolds. Suppose M and N are smooth manifolds, and suppose F M > is a smooth map. The differential of F E M is linear map between the tangent spaces:
dF T. M + Tr(m)-
The map dFn,. is computed in local coordinate charts as the Jacobian of partial derivatives. If G' N + another smooth map, then the chain rule becomes d(G F)m dGF((m) dF,,, is for any E M.
Matrix case. Suppose L Rm + R is a differentiable function. In this case, we regard the Jacobian W e I mm an m matrix:
0L
0L Surz1
8L Du
aL
0L
0L
dLw
Dwny
W
Zw
W
dur..
W
R mxn.
Purs
Pwra
W
W
W.
27

---

Published as a conference paper at ICLR 2023
where Waj the matrix coordinates.
F R + R mmx is differentiable function, regard its
Jacobian at E R as m x n matrix:
d.F11
dF
dPis
dF22
d Fa
dF
dF.
dt
E Raxm
dFm1
dFm2
dF
dt
dt
dt
where Fij R + R are the coordinates of F. Then the chain rule becomes:
)) Tr(dLF(s) dF.).
d (LoF) dt
QL
dF's
Dwig
dt
F(s)
In other words, the derivative of the composition L F at s E R the trace of the product of the matrices dLr(s) E "x and dF', E __mx"
D NEURAL NETWORKS: NON-LINEAR ACTIONS GROUP ACTIONS
In this section, we consider a non-linear action of the hidden symmetry group on the parameter space. This action has the advantage that exists for wider variety of activation functions (such as the usual sigmoid, which has no linear equivariance properties), and that it is defined for the full general linear group. However, constrast to the linear action, the non-linear action is data- dependent: the transformation of the weights and biases depends on the input data.
D.1 ROTATIONS
We first define certain orthogonal matrices. Definition D.1. For any tuple of real numbers 3 (Bi,..., define an (n + x (n++1 11) matrix R(B) as follows:
(os(33-1)
[I sin sin(3B) cos(33)
if i
(R(3))
(3:)
if = --=
if >++
where, by convention, we set Bo = Bn+1 = 0.
For example, when n 1,2, we have:
(os
sin(B1)
(3)
-sin
R(33) =
R(31,32) = sin(31) cos(
cos(31) cos(
-sin(32)
sin(B) (3) cos(B3
(in( sin
cos(3) sin(
cos(
Lemma D.2. For any tuple of real numbers B (Bi,..., Bn), have:
1. Σi cos? (3,) Ii (Bk) + sin?( =1
2. The matrix R(B) is orthogonal.
Sketch of proof. The first identity follows from a straightforward induction argument, while the proof of the second claim amounts to computation that invokes the identity of the first claim.
Proposition D.3. There is a continuous map R. R \ (0) + GLI, written ++ Ry, such that:
1. For any z E Rh {0). the first column of R2 z. Hence R.e = 2, where ej =
(1, 0, ... is the first basis vector.
28

---

Published as a conference paper at ICLR 2023
2. The operator norm of R₂ is [R [2].
3. [f|=]= then Rg is orthogonal matrix.
Proof. Let 2 E Rº {0). and let (r, ai,... an-1) ) be the (reverse) /- h-spherical coordinates of 2. Hence, r = |z| is the norm of and the i-th coordinate of = is 2j = II sin(au) cos(o:), where Oo,, = 0 by convention. Now set Rg = |z| 1[(a,..... -.,-1). Using Lemma D.2, one con- cludes that R3 is invertible with inverse |= R so that R2 has operator norm is |2| and R. is orthogonal |a| 1. is also clear that the first column of Rg is equal to 2.
We note the the matrix in D.1 has form similar, but not identical, to the Jacobian matrix for the transformation to n-spherical coordinates. Euler angles provide another way to construct a map I \ [0) + GL4,, the same properties in Proposition D.3.
D.2 NON-LINEAR ACTION: TWO-LAYER CASE
Consider two-layer network with dimension vector (m. h, n), no bias vectors, and no output acti- vation. The parameter space is Param = IR mx x Rhxxn Define the non-degenerate locus as:
(Param x - = {(U, V.x) E m R x R Vx + 0}
Let F: Param R + Rm be the extended feedforward function, taking VV to ((),))()) = (Vz). We now state and prove a more general version of Theorem 4.1.
Theorem D.4. Suppose (2)) + for all nonzero 2 R \ {0}.
1. There is an action:
GLn x (Param x R + (Param x )")
g (U,V, (URo(Va) R Roigv)) gV. x)
2. Suppose, in addition, that (0) + 0, so that a is nonzero on all of Rh. There is an action:
GL, Paaam R²) > (Param IR")
9 (U, V, x) = ( gV, x)
In both cases, the extended feedforward function invariant for this action, that is: F(g (U,V,z)) = P(U,V.x).
Proof. We first verify that the action is well-defined. In the second case, (gVv) * 0 for all (U. v.z) and hence Ra(gVa) is defined and invertible for any g GL,, For the first case, let (U, v) be in the non-degenerate locus. The non non-degenyracy condition Vz # 0 guarantees that gV2 * for all g E GL,, The hypothesis on a in turn implies that Ro(gVa) is defined and invertible for any g E GT., Hence the action is well-defined in both cases.
To check the unit axiom, observe that, when g (V Rogva Ra(va) Ro(v2) R = id, and gV = V. It follows that id (U,V, = (U, V. To check the multiplication axiom, let 91, 92 GL,. Then:
id, is the identity of GL,, we have
( Ro (Ro(g Rola
(Ro (Rotaze) Ra())) (o1 Ro(o)
It follows that (0z (U, v)) = (9192) (U, V, For the last claim, we compute:
Flg (U, v.))
(aV R-1 9V,z)
=U =
=
where the first equality follows from the definition of the action; the second from the extended feedforward function F: and the third and fourth follow from Proposition D.3.
n
29

---

Published as a conference paper at ICLR 2023
From the proof, see that key property of the matrices R₂ is that:
(gs) (a(s) (()) [gz>
(30)
This can be interpreted data- data-dependent generalization of the equivariance condition appearing in Equation (4). We emphasize that a sufficient condition for the existence of such an action that (2)) is nonzero for any nonzero RR; this is the case for usual sigmoid, hyperbolic tangent, leaky ReLU, and many other activations.
Finally, remark differential- liffereetialleeometiie interpretation of the construction of this section. One can regard as a section of the trivial bundle on R {0) with fiber R The map & + Ro((a)) R defines a GL₁, -qquivarraa structure on this bundle such that a is an equivariant section. Indeed, the action GL,, on the total space (RA \ (0)) Rh given by g (2, a) = (gz, Ro(az) Ro(z) a), and the equivariance of is precisely the condition (ox) R (z) og=))
D.3 NON-LINEAR ACTION: MULTI-LAYER CASE
We adopt the notation of Section C.2. In particular, consider neural network with L layers and widths (no, ....................... RL.) The parameter space is given by:
Param(n) = - x R R XnI-2 X... x "2x1 x R x IR x R x... x R
So for each layer i, we have matrix W; E xn and vector by R We write (W, b for choice of parameters. Fix activations 0; R - I for each i = =. L. Let
F Fg R RL
be the feedforward function corresponding to parameters = (W, b) e Param with activations Tee Taking the parameters into account, we form the extended feedforward function:
F Param X RO RT
F(0.z) 2) FO(x)
One can also define the extension of the partial feedforward function F; Param R - R as (0, 2²) + (z)( Furthermore, let Z, Param x R + R be the function defined recursively as:
i 0
Zz(6,z)
Wir bj W (4,1 +
i 2 L
We have F. o; Ze for 1,..., L, and the extended feedforward function is F 0r, Zz. Define the non-degenerate locus as:
(Param RT)" (00,z) 70 x) + for I.,
Proposition D.5. Suppose that, for i 1....,I L 1, the activation 0; IR R satisfies of (0) S (0). Then there is an action of the hidden symmetry group Gla, on the non nen-degenerate locus given by:
GL, x (Param x R")0 + (Param X R")
g (0.z) ((..W (0..)) R Rodo: 17, 100,))) 9:b:
1-1 i=1 x
Moreover, this action preserves the extended feedforward function.
Proof. The fact that the action well-defined follows from the assumption each o, and the non- degeneracy condition. The unit and multiplication axioms are shown in the same way in the proof of Theorem D.4. For the last claim, one first verifies by induction that Z:(g (0, x)) 9, Z, for 0, 1,.., L. Hence,
F'g 0, x) or (Z (0,z)) or (0.)
((,2z)
using the fact that 9r. is the identity. So the extended feedforward function is preserved under this action.
30

---

Published as a conference paper at ICLR 2023
D.4 DISCUSSION: INCREASING THE BATCH SIZE
In this section, we discuss difficulties in adopting the construction of the previous sections cases where the batch size is greater than one. Fix a batch size k. so that the feature space of the hidden layer is g4xk By abuse notation, we write a ; [R + gixk for the map applying o column-wise We say that preserves full rank matrices if o (Z) is full rank for any full-rank matrix in Z E R As final piece of notation, let (TRx C h be the subset of full rank matrices.
Lemma D.6. Suppose that k S h, and that a preserves full-rank matrices. Then there exists a map e: GL (Ra \ ((0)) - GL,, satisfying the following identities for any nonzero Z h and 9. 91, 92 E GL5::
c(id), Z) idh
(31)
(191.922c((92 Z) = c(9192, 2) dg. (Z) oggZ)
(32)
(33)
omit a proof of this lemma. A key tool the fact that, for S h, any two matrices (R) related by an element of GL This lemma implies that, for multi-layer network, if preserves full rank matrices in IR., x for each i, then there is a non-linear group action as in Proposition D.5, where the appropriate version of the non-degenerate locus is:
(Param R=x (0, Z,(0,)) R
x is of full rank for i l,..,, L 11..
However, as the following examples show, the condition that a preserves full rank matrices is not satisfied in the case of common activation functions.
Example D.1.
1. For k > 1, the column-wise application of the usual sigmoid activation does not preserve full rank matrices. For example, for = 2, take:
vtlagnto
1.3863
-0.4055
Z
00.4055
1.3863
Then det( while d2t 2.0862.
2. For ls 1, the column-wise application of hyperbolic tangent does not preserve full rank matrices. To see this, set k 1 and consider:
tanh tanh 3
tanh tanh
0.20227
0.4236
Z
0.4236
1.0986
Then det(tanh( Z)) = 0 while det (Z) 0.0432.
3. Let be a real number with 0 < s < 1. The corresponding leaky ReLU activation function is given by a22) = min(0, 2) + max(0, 2). For k > the column-wise application of leaky ReLU tangent does not preserve full rank matrices. Indeed, for = 2, set:
Z
s
11
Then det()(Z)) det. ([=
[s]
0 while det(Z) = 12--144 1
Finally, in the case l: > h, the action of GL, on full rank x matrices is not transitive. Hence, there will generally be matrix in GT., taking a (Z) (gg7), even if both are full rank.
31

---

Published as a conference paper at ICLR 2023
D.5 LIPSCHTIZ BOUNDS
Proof of Proposition 4.2. Let (U, V, x) be in the non-degenerate locus, let E GLn, and let æ1, 22 E R". Using the Lipschitz constant of and the definition of operator norms, we compute:
((0..) (a)
x2)1 - U Uo R
oiov ( x2))
n R. (V²) R
(gV
||||||||||||ar x2
n
R
Ro |llllllll x2l
[(gV=)
o (V)) n||
|| x2
[o(gVx)
The result follows.
E GRADIENT DESCENT AND DRIFTING CONSERVED QUANTITIES
While gradient flows are well approximated by gradient descent (Elkabetz. & Cohen, 2021), the conserved quantities of gradient descent are no longer conserved in gradient flow due to non- infinitesimal time steps. However, with small learning rate, we expect the change in the conserved quantities to be small. In this section, we first prove that the change of Q is bounded by the square of learning rate for two layer linear networks, then show empirically that the change Q is small for nonlinear networks.
E. CHANGE IN IN GRADIENT DESCENT (LINEAR LAYERS)
Proposition E.1. Consider the two layer linear network, where U E RR V E kR are the only parameters, and the loss function L is a function of UV. In gradient descent with learning rate т}, the change in the conserved quantity Q = Tr : - VVT at step t is bounded by
dC(t)
1Q+1 - Q21 S 172
(34)
di
Proof. Let Ug and Vg be the value of and at time in a gradient descent. The update rule
ac
OC
U = U, -U
V41 Vn - σV
(35)
aU
Consider the two layer linear reparametrization W UV.
Q: = Tr [Uद - V.V."]
Q++1 Tr [U4, U141 V441 V.T1
(36)
Substituting in U2+1 and Ve+1. expanding Qc+1. and subtracting by Qt, we have
Oc
Oc
1/ ( dC aU,
U. - nUT dc
Q+11 Q Tr
1
aU,
OU.
OU,
OL
OL
OC
0L 27
17
+ 0V,
V. + +
(37)
OV,
OV
OV,
Note that
0C aU,
U. -VCVVV = -VV V, (BC)
(LL)
(38)
aV..
and similarly
U OC Oc
VT
(39)
aU,
0V,
32

---

Published as a conference paper at ICLR 2023
Therefore, (37) simplifies
Oc
Oc
0c
Oc
Q++1 Q =
(OU,)
aU;
0V,
0v,
Oc
Oc
OC
ac
n²²
Tr
Tr
(40)
au,
au:
av,
av,
and the variation of Q in each step is bounded by the convergence rate:
0C
OC
0L
ac
Q4++ - 10-1-011= Q-1= = 12
Tr
Tr
aU,
OU
0V,
0V,
0c
Oc
OC
T
ac
$ 17
Tr
+
au,
au
0v,
0v,
dC
17
(41)
dt
E.2 EMPIRICAL OBSERVATIONS
In gradient flow, the conserved quantity Q is constant by definition. In gradient descent, Q varies with time. In order to how applicable our theoretical results in gradient descent, investi- gate the amount of variation in Q in gradient descent using two-layer neural networks.
Since Q is the difference between the two terms Si(U)
T and fz(V) 0'())' we normalize Q by the initial value of (U) and (2(V), i.e.
Ea.s /n da
o(x)
2 Tr - Eas Mvs du o(x) [(
-Ea
Q
Tr/ + E.. Voas dx a(x)
d(
and denote the amount of change in Q as
aQ( ((1) - Q(0))
(42)
We run gradient descent on two-layer networks with whitened input with the following objective
argmin V) = || Uo(
(43)
where a is the identity function, ReLU, sigmoid, or tanh. Y E 5x U 5x500 and E R 10 have random Gaussian initialization with zero mean. We repeat the gradient descent with learning rate 0, 0.01, and 0.001.
The variation AQ and loss is shown in Fig.3. The amount of change in Q is small relative to the magnitude of (U) and f2(V), indicating that conserved quantities in gradient flow are approximately conserved gradient descent. The error in Q grows with step size, as AQ(() is larger with the largest learning rate used, although it has the same magnitude those of smaller learning rates. We also observe that Q stays constant after loss converges.
F DISTRIBUTION OF Q UNDER XAVIER INITIALIZATION
We first consider a linear two-layer neural network UVX, where U E Baxh V E RAxn and X g We choose the following form of the conserved quantity:
1 Q
-
(44)
33

---

Published as a conference paper at ICLR 2023
30004
3.50
6,900
0.02
3 0834
20:5
106
10-1
20-05
60-45
Thiiring steps
Thaising staps
Thitting steps
Twiiing soeps
(a) linear
(b) ReLU
(c) tanh
(d) sigmoid
Figure 3: Dynamics of conserved quantities in GD. The amount of change in Q small relative to its magnitude, and Q converges when loss converges.
Xavier initialization keeps the variance of each layer's output the same as the variance of the input. Under Xavier initialization (Glorot & Bengio, 2010), each element in given layer is initialized independently, with mean 0 and variance equal to the inverse of the layer's input dimension:
(0. 1 hh
1
Un = =N
Vij N 0.
(45)
The expected value Q is
(46) Figure shows the distribution of Q for 2-layer linear NN with different layer dimensions. For each dimension tuples (m, h, n),, we constructed 1000 sets of parameters using Xavier initialization. The centers of the distributions of Q match Eq. (46).
B[Q] (U;)) x x + Var(Vi)) x
200
200
frequency
150-
150
frequency
150
100
100
50
50
-2
106
-103
-100
-97
-94
94
96
98
100
102
104
106
Q
(a) m. h,n 100, 100, 100
(b) m., h, 100,
200,
100
(c) m. I, 200, 100,
Figure 4: Distribution of Q for 2-layer linear NN with different layer dimensions.
Next, consider the nonlinear two-layer neural network Uo(VX), where R - R is an dlement-wise activation function. For simplicity, we assume whitened input (X = I). choose the following form of the conserved quantity:
Q = -
El
Ves dx
(I) d'(x)
(47)
aj
Figure shows the distribution of Q for 2-layer linear NN with different nonlinearities. each with 1000 sets of parameters created under Xavier initialization. The shapes of the distributions are similar that of linear networks. The value of Q usually concentrated around a small range of values. Since the range of Q is unbounded, the Xavier initialization limits the model to a small part of the global minimum.
G CONSERVED QUANTITY AND CONVERGENCE RATE
The values of conserved quantities are unchanged throughout the gradient flow. Since the conserved quantities parameterize trajectories, initializing parameters with certain conserved quantity values accelerates convergence. For two-layer linear reparametrization, Tarmoun et al. (2021) derived the explicit relation between layer imbalance and convergence rate. We derive the relation between conserved quantities and convergence rate for two example optimization problems and provide nu- merical evidence that initializing parameters with optimal conserved quantity values accelerates convergence.
34

---

Published as a conference paper at ICLR 2023
250
250
frequency
150
frequency
200
frequency
200
150
100
50
50
0
46
10000
-9950
Q
(a) ReLU
(b) tanh
(c) sigmoid
Figure 5: Distribution of Q for 2-layer linear NN with different nonlinearities, with parameter di-
mensions m n 100.
G.1 EXAMPLE I: ELLIPSE
We first show that the convergence rate is related to the conserved quantity in a toy optimization problem. Consider the following loss function with a E
(111, W2) = w + aw3
V (2wh, 2aw2)
(48)
Assuming gradient flow, dwi - w, C 22
dw2
V we -2awy
(49)
Then W1, W2 governed by the following differential equations:
w1 (1(() = e e- where Win; W2, initial values of w1 and W2. We can find conserved quantities by using ansatz Q = ( and solving VQ V for i, k. Below we use the following form of conserved quantity:
w2(() = W2 e -2at
(50)
2a
Wi
(51)
20
To show the effect of Q on the convergence rate, we fix L(0) and derive how Q affects I.(t). Let L(0) = wf, + aw3 = Lg. Let u2, continue to be an independent variable. Then wf, = Lo aw?.
Substitute in wi, the loss at time l is
I(() wi (1)² + aw2(1)² (To aw², )e awz e 1at
(52)
and Q becomes
w
(Lo aw2. )a
Q
(53)
22
wy
The derivative of in the direction of Q
dL) dw2. dl(()
dQ
Lit( dw2, dQ dwzo
dw2
(aLLo awz,)" (0-1(-2aw,, )wz
22 (Lo aw3 )a
-1
- 1-2aw2 + 2aw2, lut)
(-2aw2 + 2awzze--un lat) w²
al Lo aw3, )a-1( 20ww JwI, 2w22 Lo - aw?,
2aw2z (e (e-dut -1)
(54)
2w2 (Lo awz )a-1 a²w3, - (Lo aw?,))
In general, L(t + O. meaning that the loss at time t depends on Q. Since we have fixed the initial loss, the convergence rate L(t) L(0) also depends on Q. Special cases where Lt include a = (circle), a = (collapsed dimension), and certain initializations such as Wa, = (local maximum of gradient magnitude).
35

---

Published as a conference paper at ICLR 2023
G.2 EXAMPLE 2: RADIAL ACTIVATION FUNCTIONS
In this example, we find the conserved quantities and their relation with convergence rate for two- layer reparametrization with radial activation functions under spectral initialization. Define radial function g: m mmxx as
g( in ([Wa)) Waj.
(55)
where |W:| = x w is the norm of the th row of W, and h : R - R outputs a scalar.
Consider the following objective:
argmin v V) = - -Ug
(56)
with spectral initializations
Uo 2U Vo VV
where 4 come from the singular value decomposition = &h V and Un, V, are random diagonal matrices.
Proposition G.1. Under the gradient flow U = Vu and V = vv the following quantity is an invariant:
I E Q = 2
g(x) g'(x)
(57)
Proof. Since is radial function on rows and 7 is an orthogonal matrix, g(V -²)= g(V Dy? With spectral initialization, the loss function can be reduced only involving diagonal matrices:
C Y Ug(V") >|
T
vU "Ug[(
T (V 1
1 () ---U((()) w/'
2 Σ Ug(V")) >|
(58)
Since V is a diagonal matrix, g is now an element wise function on V. Let W = Ug(V"). The gradients for U and V are
0C
= WLCgVV)
OU
OL
0V = U o
(59)
where g'(x) dg(x)/dz is the derivative of the nonlinearity. Additionally, since L does not depend on and V,
ac Ac 04
0
(60)
0V
Since the rows of Φ, v are orthogonal,
dC Oc
T TT
OU O
Dc Oc
TT (V ""'
(61)
0V Ae
36

---

Published as a conference paper at ICLR 2023
& and v are not changed in gradient flow, aq au
aQ tT and 0Q aQ T² Define inner product
av oF on matrices as (X, Y) = -T][X"']. For Q to be a conserved quantity, we need (VL, VQ) = 0:
ac 0Q
Oc aQ
(VL, VQ) OT OU
+ OV av
(VW TT TT iQ 4 T) + (VV W L 3
(V) DT 0Q
0V vT)
0U
=(VwCo(V) 0Q
aQ
((V W U
..
= Tr QV +U V C(v
0
(62)
Following the same procedure as for elementwise functions, to have Q which satisfies (62) it is sufficient have
aQ f(U,
aQ
0v g'(V)aj --((T,V)(V)) f(U.V) E
(63)
a0
For simplicity, [(U, V) Then, (63) is satisfied by
Q 1 -1 2
ES
Vi: dx
g(x) g((x)
(64)
Tarmoun et al. (2021) shows that the conserved quantity Q appears as a term in the convergence rate of the matrix factorization gradient flow. We observe a similar relationship between Q and convergence rate when the loss function is augmented with a radial activation function, shown in the following proposition.
Proposition G.2. Consider the funcciovv and spectral initialization defined in Proposition G.I. Lel h (la)) = -W, and X = UV(V = xΣx V2" Then, the eigencomponent of X approaches the corresponding eigencomponent of at a rate of
X 1o of) (o,x² + 1)3
(65)
where oN diag(Ex); )i of = diag(E, ), and X; = U3 + V2 are conserved quantities.
Proof. Similar to Tarmoun et al. (2021), components can be decoupled, and we have a set of differ- ential equations scalars:
ū. = [o - ū.g (T)
ग. = joy - T.o(T.)) dg(
(66)
dv:
We also have
dg dv; ((T,) do, dl
dg((:)) 2 dii
JoY (..((T)){T..
(67)
Let a:x u.g( Then
0x¹ = (. (T.(TT))
dg((s)) do;
27
= [oo - V.g(T.)) g(T.)2 + op
(68)
Since V is diagonal matrix, g is now an element wise function on V. Specifically, g(v,) According to Proposition G.1, the following quantity is invariant:
2 T1 -f dx g(x)
-
T dr
0
(69)
g'(x)
U.
2
2
37

---

Published as a conference paper at ICLR 2023
Since any function of the invariant is also invariant, will use the following form:
Q Q=U'U D-U'U+V"V vVV,
(70)
and define
X Qii = uz D
(71)
Using the g that we defined,
0 u.g(v:) U.y."
(72)
In order to relate x and Q, we first write T7 and 3. as functions of ox ad Q using (71) and (72):
X x2
A
(73)
Then, substitute u. U.. (T;)), and da(ws) de,
into (68), and we have
dd (T+)) du,
27
0x¹ lo; u.)(
g(T,)² + up
To [.g(T.)]
+ t 2)
Jo [.g(T,)]
[() -1 पp(TT)) ?
A
1o L:io
ox +1
qj.² +1
/; +1
oxx +1
x2 (op²² + 1)
To Lkio
A
A
0xx² 20x²² +1
To
X
(o - o,²² (o, + +1)²
(74)
Proposition G.2 relates the rate of change in parameters o. and the conserved quantity A; To get a more explicit expression of how A; affects convergence rate, we will derive bound for la a which describes the distance between trainable parameters to their desired value.
Proposition G.3. The difference between the singular values U UVTT)) and Y is bounded by
1ox - -o s 10x (0) of le x
(75)
Proof. Note that
oxx xx= 10- - -o ,ox² + +)
1 [ (0) -ox)
(76)
Consider the following two differential equations, with same initialization a(0) b(0):
1
A (a a)(a² +1)
1
b = A
(b-
(77)
In these equations, both a and b moves from a()) = b(0) to a monotonically. Since à 2 b at every b, a will always be closer than b does. We can explicitly solve for b, which yields
38

---

Published as a conference paper at ICLR 2023
b(t) + (b(0) dee x. Then the distance between and is al [6(0) ooe- x Using lb σ we can bound la of:
la ol ol S ol [b(0) ale
(78)
Therefore,
lo." - - 10,² (0) le
(79)
n
Since À is a conserved quantity, its value set initialization remains unchanged throughout the gradient flow. Therefore, we are able to optimize the convergence rate by choosing a favorable value for À at initialization. In this example, smaller A₂'s lead to faster convergence.
G.3 EXPERIMENTS
We compare the convergence rate of two-layer networks initialized with different Q values. We run gradient descent on two-layer networks with whitened input with the following objective
argminy, (CC, V) 1| (o(V") [[
(80)
where a is the identity function, ReL.U, sigmoid, or tanh. Matrices Y E R5x10 U E R5x50 and V IR 50 have random Gaussian initialization with zero mean. We repeat the gradient descent with learning rate 0.1, 0.01, and 0.001. The learning rate is set to 10 3 as we do not observe significant changes in the shape of learning curves at smaller learning rates, U and V are initialized with different variance, which leads to different initial values of Q.
102
102
102
107
228
107
20"
1944
941
10--
10-1
10-2
10-2
De-c5 Traning steps
de+55
2e-05 Traning steps
2e-00
de
4e-005
86-05
The ning steps
The ming steps
(a) linear
(b) ReLU
(c) tanh
(d) sigmoid
Figure 6: Training curves of two-layer networks initialized with different Q The value of Q affects convergence rate.
As shown in 6ig.6, the number of steps required for the loss curves to drop to near convergence level correlated with Q in both linear and clement-wise nonlinear networks. This result provides empirical evidence that initializing parameters with optimal values for Q accelerates convergence.
We then demonstrate the effect of conserved quantity values on the convergence rate of radial neural networks. Fig.7 shows the training curve for loss function defined in Proposition G.2. We initialize parameters U E 55x5 and V E R 10x5 with 4 different values of Q and the learning rate is set to 10 10-5, As predicted in Eq. 75, convergence is faster when Q T + VTV] is small.
H CONSERVED QUANTITY AND GENERALIZATION ABILITY
Conserved quantities parameterize the minimum of neural networks and are related to the eigenval- ues of the Hessian at minimum. Recent theory and empirical studies suggest that sharp minimum do not generalize well (Hochreiter & Schmidhuber, 1997; Keskar al., 2017; Petzka et al., 2021). I). Explicitly searching for flat minimum has been shown to improve generalization bounds and model performance (Chaudhari et al., 2017; Foret et al., 2020; Kim et al., 2022). We derive their relation- ship for the simplest two-layer network, and show empircally that conserved quantity values affect sharpness. Like convergence rate, a systematic study of the relationship between conserved quantity and generalization ability of the solution is interesting future direction.
39

---

Published as a conference paper at ICLR 2023
10:1
10"1-
geo 206
Q-20..6
Qm2 06eee 33
10-10
0.2
0.6
Training steps
Figure 7: Training curve for the loss function defined in Proposition G.2. Smaller value of Q = TUUTU +V at initialization leads faster convergence.
Q= -3 Q 0 3 minimum
20
15 L 1.0
6.0
U
V
Figure 8: Gradient flow for L(U, V)
1
UV where U. R, Y 2, and X 1.
Trajectories corresponding to different values of Q intersect the minima at different points.
H.1 EXAMPLE: TWO-LAYER LINEAR NETWORK WITH 1D PARAMETERS
We again consider the two-layer linear network with loss L = = - UV For simplicity, we work with one dimensional parameters U, V E I and assume X = = 1 in this example. We show that at the point to which the gradient flow converges, the eigenvalues of the Hessian are related to the value of the conserved quantity.
The gradients and Hessian of L are
VC (Y VX (Y - UVXJUX
x2x2
-X + 2
H -YX 2VV
(81)
U2
At the minima, U, V are related by UVX Y. Recall that Q = U2 22 is a conserved quantity. From the above two equations, we can write U.V as functions of Q. Taking the solution U
V 41 + + V }( Q 2Q² 4) and substitute in X = Y 1, we have
1 Q + Q2 +4)
H =
(82)
1(Q + Q + 4)
and the eigenvalues of H. are
\1 = 0,
Ae = 2 +4.
(83)
We have shown that Q is related the eigenvalues of the Hessian at the minimum. Since the eigenval- ues determines the curvature, Q also determines the sharpness of the minimum, which is believed to related to model's generalization ability. The result in this example can also be observed in Figure 1, where the minimum of the Q 0 trajectory lies the least sharp point of the loss valley.
40

---

Published as a conference paper at ICLR 2023
H.2 EXPERIMENTS: TWO-LAYER NETWORKS
The goal of this section is to explore the relation between Q and the sharpness of the trained model. We measure sharpness by the magnitude of the eigenvalues of the Hessian, which related the curvature at the minima. We use the same loss function (80) in Section G.3. The parameters are U E IR 10 and V E R25500 each initialized with zero mean and various standard deviations that lead to different Q's. We first train the models using gradient descent. We then use the vectorized parameters the trained model to compute the eigenvalues of the Hessian.
The linear model extends the example in Section H.1 to higher dimension parameter spaces. 700 out of the 750 eigenvalues are around 0 (with magnitude < 10-3), which verifies the dimension of the minima in Proposition C.3. After removing the small eigenvalues, the center of the eigenvalue distri- bution correlates positively with the value of Q (Figure 9()). In models with nonlinear activations, Q is still related to eigenvalue distributions, although the relations seem to be more complicated.
0-1.2
350
Frequacy
15
0-122
0-493
Frequancy
20
G-263
D-237
Frequancy
0=-23
0-225
Q--213
Qm-222
16.
0-134
15
20.
150 Eigenvalues Hessian
30
Egenvaluss Hessian
Eigenvalues of Hessian
Eggenvauues Hessian
(a) linear
(b) ReLU
(c) tanh
(d) sigmoid
Figure 9: Eigenvalues of the Hessian from trained models initialized with different conserved quan- tity values (Q). The distribution of the eigenvalues and the value of Q appear to be related.
I ENSEMBLE MODELS
In neural networks, the optima of the loss functions are connected by curves volumes, on which the loss is almost constant (Freeman Bruna, 2017; Garipov et al., 2018; Draxler et al., 2018; Benton et al., 2021; Izmailov et al., 2018). Various algorithms have been proposed to find these low- cost curves, which provides a low-cost way to create an ensemble of models from a single trained propose
model. Using our group actions, values. We show that even with stochasticity in the data, the loss is approximately unchanged under the group action (Appendix I). This provides an efficient alternative to build ensemble models, since the transformation only requires random elements in the symmetry group, without any searching or additional optimization.
way of constructing models with similar loss
We implement our group actions by modifying the activation function between two consecutive layers. Let X be the output of the previous layer. The group action on the weights U, is
g (U,V) = (
(84)
where <(g, H) = - The new activation implements the symmetry group action
(H(H)- + Un(g, (gH)
(85)
by wrapping the transformations around an activation function o'(x) T(g, so that UH'(H)= Un(g, (gHH.
We test the group action on CIFAR-10. The model contains a convolution layer with kernel size 3, followed by a max pooling, fully connected layer, a leaky ReLU activation, and another fully connected layer. The group action is the last two fully connected layers. After training single model, we create transformed models using g = / + eMM, where M E R232332 is a random matrix and controls the magnitude of movement in the parameter space. We then use the mode of the transformed models' prediction as the final output.
We compare the ensemble formed by group actions four ensembles formed by various random transformation. Let g = 1 + eM. The random baselines are:
'group': VU (Um(g, H).V). This is the model created by group actions.
41

---

Published as a conference paper at ICLR 2023
g- (U,V) (Ug
random': (U + (Ug',gV) vV), where g' = I++ =D and D is random diagonal matrix.
'hufffee:: (U, V) = (U+'(g H), where N (g H) is constructed by randomly shuf- fling m(g, H).
'interpolated permute" 'perm_interp': (U,V) (U [+
1+5(1+5) V).
I+E
where S E 32 is random permutation matrix.
Figure 10 shows the accuracy of the ensembles compared to single models. The ensemble formed by group actions preserves the model accuracy for small e and has smaller accuracy drop at larger e. The ensemble model also improves robustness against Fast Gradient Signed Method (FGSM) attacks (Figure 11). Under FGSM attacks with various strength, the ensemble model created using group actions consistently performs better than the baselines with random transformations. However, the same improvement is not observed under Projected Gradient Descent (PGD) attacks.
Intereedded
800
0.001
800
0301
-00
Figure 10: Change in accuracy compared to the original single model when using the ensemble model and 4 baselines. The red color indicates degradation model performance. The ensemble created by group actions has similar loss values when small.
0.6
original
original
0.5
group
group
Accuracy
g
Accuracy
0.4
g"
perm_inter interp
0.3
perm interp
0.2
0.1
0.0
0.000
0.004
0.008
0.012
0
2
3
FGSM epsilon
PGD iterations
(a) FGSM
(b) PGD
Figure 11: Adversarial attacks the original model and the ensemble models with various strengths. In FGSM, the group ensemble model improves robustness. In PGD, the ensemble has negligible effects on robustness.
42


Reviews:
[{"review_id": "kN6Q-PtxFx5", "reviewer": "Reviewer_J3kj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates the role of symmetries in multi-layer perceptrons,\nputting forward the idea of data-dependent symmetries.\nConservation laws stemming from Noether's theorem are discussed.\n", "summary_of_the_review": "The paper should be mostly rewritten and provide a more clear focus on the results and their implications.\n", "strength_and_weaknesses": "\nI had a very hard time reading the paper. I find the organization\nof the paper quite confusing. A lot of mathematical formalism\nis squeezed in every page while giving away very little in terms of \nexplanations and intuions. Much of the paper revists classical concepts from group theory and their physical implications through Noether's theorem,\nwith  novel contributions seemingly contained in Section 1.5\nand possibly Section 2. It is hard to asses the relevance of such contributions.\n\nThe equivariance property they propose in eq. 12 and 13 seems useful\nto produce a new set of parameters with approximately the same loss as the original ones\nwhen observing a new batch of data that is close (up to a permutation) to the original data. In practice though I'm not sure how useful this is, and how \"close\" the data should be, since it has not been experimentally tested in the manuscript.\n\nThe analysis in Section 2 of the conserved quantities along the dynamics is also interesting, but it's not clear how much of it is original and how much is relevant. The experiments are not properly explained in the main text.\n\nIt could be the case that I didn't understand the main points of the paper, \nwhich in any case calls for a much better rewrite since I can easily read through\nsimilar papers such as e.g. \"Noether’s Learning Dynamics\" arxiv:2105.02716.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hhV7ano6M2", "reviewer": "Reviewer_M5tH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper provides an analysis of symmetries inherent to the architecture of certain models (overall the study is centred in neural-networks-like predictors), as well as the loss and data. Link to the set of local minimizers is also made. Finally, it is also shown that these symmetries have a corresponding echo in conserved quantities during gradient flow training.", "summary_of_the_review": "Overall, even if the paper could be very nice I think that I feel disappointed when it comes to summarize the results. Beyond learning a nice theory of Lie invariant, I do not know what I have really learnt after reading this. The novelty and promising application to data-dependant symmetry, in particular, is a bit disappointing. \n\nI will be very glad to largely increase my score if the authors answer my concerns and questions.", "strength_and_weaknesses": "## **Strength**\n\nThe first real strength of the paper is that, despite the technicality of the theory of invariance, the paper is pretty clear and easy to follow (despite some paragraphs and typos -- see *Weaknesses*). Overall, at least at the beginning of the paper, there is a real care about pedagogy, and, being personally not familiar with this theory, I felt that the paper is really understandable from this perspective.\n\nFurthermore, beyond this point of view, two results appeared particularly interesting, at least to me:\n- They manage to recover, with their general framework, standard invariant already derived in precedent works that were simply seen as \"tricks\"\n- The fact that the authors try to tackle data-dependant symmetries\n\n---------------------------------------------\n\n## **Weaknesses**\n\nHere is a list of potential weaknesses that I personally felt when reading the paper: \n\n- First, even if the authors try to tackle the case of potential data-dependant non-linear symmetries, they do not find any application of this perpective for neural networks. In fact, the only conclusive result they have on these is the known balance property, but this *only* comes from a function symmetry and is not data-related. Do the authors have in mind an example where some data invariance is of paramount importance ? (I have a more detailed question about this in the next box).\n- In fact, going further, the different paragraphs of Subsection 1.5 are really fuzzy and there is a lack of details in the calculations. Propositions and facts are inline without emphasis between them so that it is very diffiicult to follow. Furthermore, it appeared to me (maybe I am wrong), that some of the derivations are very fast and this prevents for a proper easy reading of this *very technical* section. Examples of these are e.g.: (i) second line of the **General equivariance** paragraph $c(g_1, g_2 Z) c(g_1, g_2 Z) = c(g_1g_2, Z)$, what is a typical $c$? What does it mean \"with physical interpretation\"? why is this property called \"equivariance\"? This paragraph lacks of lot of details (both in the interpretation and in calculation) to be followed. (ii) Second line of **Using nonlinear transformed [...]** paragraph: do really the authors want to write: $g \\cdot (U, V, X) = (\\tilde{U} (g, VX), gV, X)$, and in this case, what is $\\tilde{U}$? In this paragraph, it is concluded that at permutation + small deformation of the data set the function should be almost the same: why should we care about this perturbative results? I have in mind that the assumption on the data for generalisation is more that they come iid from the same distribution not that they are close to each other in distance? (iii) The last paragraph of this section on **adapting models** is very hard to follow, notations are not properly introduced, e.g. in $Z' = gZP$, what is $g$? Why should we look for $U'\\sigma(VX') = U\\sigma(VX)P$? I am sure there are some calculations that permit to go from one line to the others and things are clear in the minds of the authors, but I felt in was impossible to follow this paragraph. Maybe a real mathematical Proposition (proved in Appendix) can clarify everything here: e.g. *Assume X' is such that $\\exists g \\in G$ such that for all $V$, [...]*.\n- As far as I understood, the authors are not the first to consider symmetries to explain invariants of the dynamics. Comparison with results of previous works would be welcome to understand how it articulates with theses previous works. I am sure that all the propositions of Section 1 are well-known facts for example. \n- Minor issues: Eq (6), the gradient is seen as a linear form in this article it seems because operations with matrices are left operations. It can be troubling so it needs to be said. Typo page 6, paragraph 3: $\\sigma(a) - \\sigma(b)$. Paragraph **need for data-dependant symmetries**, isnt this $\\mathcal{L}(\\theta', X') = \\mathcal{L}(\\theta, X') $ (and not $\\mathcal{L}(\\theta, X)$). After proposition 1.6, $F(x)$ should be $U\\sigma (Vx)$. After formula (10), the infinitesimal form of $\\sigma(gz)$ ... Like previously the formula (11) is weird as $\\langle Mz, d\\sigma_z \\rangle$ is a vector and the $\\langle \\cdot, \\cdot \\rangle$ usually indicate a scalar product. As far as I understand, the authors may come from a theoretical physics community where such notations may be usual.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "MqmfDYEuWt2", "reviewer": "Reviewer_z99y", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper seeks to better understand theoretically symmetries in the loss landscape of neural networks. It first uses group theory to formulate a more general notion of parameter space symmetries, beyond permutations. The authors then motivate and present a class of nonlinear, data-dependent symmetries and a procedure for deriving conserved qualities associated with the symmetries. They show that these conserved qualities can be used to parametrize symmetric flat directions and show numerical results on two layer neural networks.", "summary_of_the_review": "The paper offers an interesting perspective on understanding neural network loss landscapes through ideas from group theory and physics. It would be strengthened if the empirical results could be reproduced beyond a toy dataset.\n\ncontext for review: I'm familiar with/written empirical deep learning papers but less capable of assessing the significance of the theoretical contributions in this work. I skimmed the appendix/some related works in writing this review.", "strength_and_weaknesses": "### Strengths\n- The characterization of symmetries is a novel perspective and sheds some light on a deep learning phenomena of interest (e.g. ensembles formed along flat minima have been shown to generalize better). The paper describes detailed analysis wider class of data-dependent symmetries and how they relate to conserved qualities in gradient flow. \n- The empirical results are interesting and support the theory. The conserved quality Q affects the convergence rate and distribution of the eigenvalues of the Hessian. The group action allows for an ensemble without retraining, which has potential in improving robustness to adversarial attacks.\n- Related work is discussed well throughout the paper.\n\n### Weaknesses\n- It would support the theory to run experiments on a non-toy dataset. Currently, they are done with small randomly generated matrices. If the same conclusions can hold for MNIST/Cifar, these results would be more relevant for practitioners. I believe experiments relating symmetries to OOD data would also strengthen the discussion in section 1.5.\n- The paper is long (56 pages with appendix) and has many different examples/derivations. I'm not sure a conference format such as ICLR is the correct venue for such a submission.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4LsKE44ILn", "reviewer": "Reviewer_FC6G", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the parameter space symmetries induced by equivariances in the activation function as well as data dependent nonlinear symmetries.  Furthermore they formalize how these symmetries lead to conserved quantities during gradient flow.", "summary_of_the_review": "**Summary**\n\nThis paper offers a useful formalization of how equivariances lead to parameter space symmetries which in turn lead to conserved quantities of gradient flow.  They offer an extensive tutorial in the appendix.  I believe the theoretical community will appreciate this work.", "strength_and_weaknesses": "**Strengths**\n\nThe paper introduces a general framework to relate symmetries of parameter space to conserved quantities of gradient flow, which is something I was hoping to see.  They make extensive effort in the appendix to catalogue applications of their theory to common activation functions and give an extended tutorial of their theory.\n\n**Weaknesses**\n\nI think the data dependent symmetries are the most interesting component, and the work could be enhanced by more experiments or discussion that give insight into what these data dependent symmetries look like in some settings.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster