Remove Symmetries to Control Model Expressivity and Improve Optimization
Authors: Liu Ziyin
Yizhou Xu
Isaac L. Chuang
Venue: ICLR 2025 Poster (2025)
Keywords: model capacity
symmetry
optimization
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: When symmetry is present in the loss function, the model is likely to be trapped in a low-capacity state that is sometimes known as a ``collapse." Being trapped in these low-capacity states can be a major obstacle to training across many scenarios where deep learning technology is applied. We first prove two concrete mechanisms through which symmetries lead to reduced capacities and ignored features during training and inference. We then propose a simple and theoretically justified algorithm, \textit{syre}, to remove almost all symmetry-induced low-capacity states in neural networks. When this type of entrapment is especially a concern, removing symmetries with the proposed method is shown to correlate well with improved optimization or performance. A remarkable merit of the proposed method is that it is model-agnostic and does not require any knowledge of the symmetry.

Published as a conference paper at ICLR 2025
REMOVE S SYMMETRIES TO CONTROL MODEL EXPRES- SIVITY AND IMPROVE OPTIMIZATION
Liu Ziyin 1,2,, Yizhou Xu Isaac Chuang!* 1.4
Research Laboratory of Electronics. Massachusetts Institute of Technology
2 Physics & Informatics Laboratories, NTT Research
3 Computer and Communication Sciences, École Polytechnique Fédérale de Lausanne Depariment of Physics, Massachusetts Institute of TeTchnologyy
ABSTRACT
When symmetry is present in the loss function, the model likely to be trapped in a low-capacity state that is sometimes known as a "collapse." Being trapped in these low-capacity states can be a major obstacle to training across many scenarios where deep learning technology is applied. We first prove two concrete mecha- nisms through which symmetries lead reduced capacities and ignored features during training and inference. We then propose a simple and theoretically justi- fied algorithm, syre, to remove almost all symmetry-induced low-capacity states neural networks. When this type of entrapment is especially a concern, removing symmetries with the proposed method is shown to correlate well with improved optimization performance. A remarkable merit of the proposed method is that it is model -agnotig and does not require any knowledge of the symmetry.
INTRODUCTION
The unprecedented scale and complexity of modern neural networks, which contain a vast number of neurons and connections, inherently introduce a high degree of redundancy in model parameters. This complexity the architecture and the design of loss functions often implies that the loss func- tions are invariant to various hidden and nonlinear transformations of the model parameters. These invariant transformations, or "symmetries," " the loss function have been extensively documented in the literature, with common examples including permutation, rescaling, scale, and rotation symme- tries Simsek et al. 2021 Entezari et al. [021 Dinh et al. 2017 Neyshabur et al. 2014 Tibshirani
202
Zhao et al. 2023 Godfrey et al. 2022 Ziyin et al. 2024).
unifying perspective to understand how these symmetries affect learning the framework of re- flection symmetries (Ziyin 2024 [ A per-sample loss function possesses P-reflection symmetry for all 0, and all data points x,
(I) The solutions where 0 = PI are the symmetric solutions with respect to P. It has been shown that almost all models contain quite a few reflection symmetries. Permutation, rescaling, scale, and rotation symmetries all imply the existence of one or multiple reflection symmetries in the loss.
- [(11-P))) =))
Recent literature has shown that symmetries in the loss function of neural networks often lead the formation of low-capacity saddle points within the loss landscape (Fukumizu) 1996: Li et al.. 2019). These saddle points are located at the symmetric solutions and often possess a lower capacity than the minimizers of the loss. When model encounters these saddle points during training, the model parameters are not only slow to escape them but also attracted to these solutions because these the gradient noise also vanish close these saddles (Chen et al. 2023). Essentially, the model's learning process stagnates, and it fails to achieve optimal performance due to reduced capacity. However, while many works have characterized the dynamical properties of training algorithms close to symmetric solutions, no methods are known to enable full escape from them.
Because these low-capacity saddles are created by symmetries, we propose method to explicitly remove these symmetries from the loss functions of neural networks. The method we propose is
"quual contribution.
"Fssentially this is because the common symmetries in deep learning all contain Zg subgroup.

---

Published as a conference paper at ICLR 2025
theoretically justified and only takes one line of code to implement. By removing these symmetries, our method allows neural networks to explore a more diverse set of parameter spaces and access more expressive solutions. The main contributions of this work are:
1. We show how discrete symmetries in the model can severely limit the expressivity of neural networks in the form commonly known "collapses" (Section[4)
2. We propose a simple method that provably removes almost all symmetries in neural networks, without having any knowledge of the symmetry (Section(5):
3. We apply the method to solve a broad range of practical problems where symmelry-impaired training can be major concern (Section
We introduce the notations and problem setting for our work in the next section. Closely related works are discussed in Section All proof's and experimental details are deferred to the appendix.
2 DISCRETE SYMMETRIES IN NEURAL NETWORKS
Notation. For a matrix A, we use A+ to denote the pseudo-inverse of A. For groups U and G, U a G denotes that U is subgroup of G. For vector 0 and matrix D, ||@1] == 8 DO is the norm of 8 with respect to D. o denotes the glement-wise product between vectors.
Let f(0,z) be a function of the model parameters @ and input data point For example, f could either be a sample-wise loss function l or the model itself. Whenever f satisfies the following condition, we say that o has the P-reflection symmetry (general symmetry groups are dealt with in Theorem 5 in Section 55.
Definition 1. Let P be a projection matrix and O' be a point. f is said to have the (0', P)-reftection symmetry for all x and 0, (1) f(0 0', f 2P)0 + 0', x)) and (2) P0' 0'.
The second condition is due to the fact that there is a redundancy in the choice of 0' when 0' # O. Requiring P8' 6' removes this redundancy and makes the choice of 8' unique. Since every pro- jection matrix can be written as product of (full-rank or low-rank) matrix O with orthonormal columns, one can write P O0" and refer to this symmetry as an symmetry. In common deep learning scenarios, it is almost always the case that 0' 0 (for example, this holds for the common cases of rescaling symmetries, (double) rotation symmetries, and permutation symmetries, see The- orem 2-4 of Ziyin )).24)) A consequence of 0' = 0 is that the symmetric projection P0 of any 0 always has a smaller norm than 9: thus, a symmetric solution is coupled to the solutions of weight decay, which also favors small-norm solutions. As example of reflection symmetry, consider simple tanh network f(0, x>) = 6, tanh(02z), The model output is invariant to a simultaneous sign flip of 0, and 02. This corresponds P= (11,0), (0, 1)). . The symmetric solutions correspond to the trivial state where 0₃ = 02 = 0.
reflection symmetry whose projection matrix is the identity
To be more general, we allow 0' to be nonzero to generalize the theory and method generic hyperplanes since the purpose of the method is to remove all reflection symmetries that may be hidden difficult to enumerate. Let us first establish some basic properties of loss function with P-reflection, to gain some intuition (again, proofs are in the appendix).
Proposition 1. Let f have the (#', P)-symmetry and let f'(0) = f( + +'). Then, for any 8 such that P8 = 0', f(0 8*) = f[[[- 2P) + f*), and (2) f' has the (0' -y-6t ft P) symmetry.
Therefore, requiring P0' = 6 is without loss of generality: it only reduces different manifestations of the symmetry to a unique one and simply shifting the function will not remove any symmetry. This proposition emphasizes the subtle difficulty in removing a symmetry.
RELATED WORKS
One closely related work that Ziyin 2024). which shows that every reflection symmetry in the model leads to a low-capacity solution that is favored when weight decay is used. This is because the minimizer of the weight decay is coupled with stationary points of the reflection symmetries the projection of any parameter to symmetric subspace always decreases the norm of the param- eter, and is thus energetically preferred by weight decay. Our work develops a method to decouple symmetries and weight decay, thus avoiding collapsing into low-capacity states during training. Be- sides weight decay, an alternative mechanism for this type of capacity loss is gradient- noise induced collapse, which happens when the learning rate- batchsize ratio is high (Chen et al. 2023).
2

---

Published as a conference paper at ICLR 2025
Contemporarilyy Lim al. (2024) empirically explores how removing symmetries can benefit neural network training and suggests a heuristic for removing symmetries by hold a fraction of the weights unchanged during training. However, the proposed method is only proved to work for explicit permutation symmetries in fully connected layers. This is particularly a problem because most of the symmetries in nonlinear systems are unknown and hidden (Cariglia 2014 Frolov et al. (017). In sharp contrast, the technique proposed in our work is both cechitcctur--ndeeeenee and symmetry- agnostic and provably removes all known and unknown discrete symmetries the loss.
SYMMETRY IMPAIRS MODEL CAPACITY
We first show that reflection symmetry directly affects the model capacity. For simplicity, we let 8' 0 for all symmetries. Let fxx 0) & E be Txylor-expandable model that contains P-reflection symmetry. Let A = - On- Then, close to any symmetric point 0g (any 0n for which P0, = 0), for all x. Ziyin 2024 showed that
f(z,0) faz 0o) -of(x, (o) P))[[- ATHH())) P 0( (2)
Symmetry subspace
Symmetry brokenn subspace
where (x) is the Hessian matrix of f. An important feature is that the symmetry subspace is a generic expansion where both odd and even terms are present, and the first order term does not vanish in general. In contrast, in the symmetry-broken
subspace, all odd-order terms in the expansion vanish, and the leading order term is the second order. This implies that close a symmetric solution, escaping from it will be slow, and if at the symmetric solution, it is impossible for gradient descent to leave The effect of this entrapment can be quantified by the following two propositions.
Proposition 2. (Symmetry removes feature.) Let f have the P-symmetry, and 0 be intialized Og such that P6, = 0. Then, the kernalized model, g(2, 0) = limx--o( [(x,, 20 0o) (o),,)), converges
0* = A* E Vof(E, v(())
(3)
under GD for sufficiently small learning rate. Here A= (I- P) and A denotes the Moore-Penrose inverse of A.
(T)
This means that in the kernel regime being at sym- metric solution implies that the feature kernel features are being masked by the projection matrix:
0.10
0.10
0.06
0.06
Vof(x, - (I P),of(1,8o)
(4)
0.04
0.04
and learning can only happen given these masks. This implies that the model is not using the full feature space that is available it.
0.02
0.02
0.00
0.00
0
500
500
Proposition 3. (Symmetry reduces parameter dimen- sion.) Let f have the P-symmetry, and 0 e Rd be in- tialized at Go such that Pf, O. Then, for all time steps t under GD or SGD, there exists model fa(x, (xx and sequence of parameters Of such that for all x,
Figure 1: Training loss & as function of the iteration for fully connected network MNIST. Starting from low-capacity state, vanilla neural networks are trapped un- der SGD or Adam training (left). Blacker lines correspond to izations, where highet-capacity initial- neurons away from the permutation -ymmiiion--mmtiiiicccc state. When the symmetrics are removed, the capacity of the initialization no longer affects the solution found at the end of the training (right).
('xx000 (x,
(5)
where
The existence of this type of dimension reduction when symmetry present has also been noticed by previous works in case of permutation symmetry (Simsek et al.) 202I). Intuitively, the above two results follow from the fact that the symmetric subspaces of re- flection symmetries (and any general discrete symmetries) are linear subspace, and so gradient descent (and thus gradient flow) cannot take the model away from it, despite the discretization er- ror. This means that, essentially, the model is identical to model with strictly smaller dimension
Technicallyy, this is the lazy training limit Chizat al. 2018

---

Published as a conference paper at ICLR 2025
throughout its training, matter whether the training is through GD SGD. When there mul- tiple symmetries, there is a compounding effect. See Figure for an illustration. We initialize a two-layer ReLU neural network identical (corresponding to the symmetric states of the permutation symmetry) and train with and without removing the symmetries. We see that when the symmetries proposed in the next section), the model is no longer stuck at these neuron-collapsed solutions. Also, see Section C for the evolution of NTK and gradient covariance during training.
low-capacity state where a fraction of the hidden neurons are
removed (with the method
REMOVING SYMMETRY WITH STATIC BIAS
Next, we prove that a simple algorithm that involves almost no modification to any deep learning training pipeline can remove almost all such symmetries from the loss without creating new ones. From this section onward, we will consider the case where the function under consideration is the loss function (a per-batch loss or its expectation); f f=
5.1 COUNTABLE S SYMMETRIES
We seek an algorithm that eliminates the reflection symmetries from the loss function t. This sub- section will show that when the number of reflection symmetries in the loss function is countable, one can completely remove them using a simple technique. The symmetries are required to have the following property and the loss function is assumed to obey assumption] [
Property 1. (Enumeratability) There exists countable set of pairs of projection matrices and biases S = P.)) such that E(0,z) has the 0% -centric P.-reflection symmetry for all i. In addition, does not have any (0*, P) symmetry for (0, P) S...
Assumption 1. There only exists countably many pairs (Co. f) such that g(x) = {(0,2) - cu0 contains a O-centric P symmetry, where we require Peo = on and PÔ = 6.
PŪ=
This assumption is satisfied by common neural networks with standard activations. The main pur- pose of this assumption is to rule out the pathological of a linear quadratic deterministic objective, which never appears in practice or for which symmetry is not a concern
For symmetry removal, we propose utilize the following alternative loss function. Let 6g be drawn from a Gaussian distribution with variance du and be the original loss function:
(6) y is nothing but the standard weight decay. We will that using static bias along with weight
(0, E(O + 0o) + y
decay is essential for the method to work. We find that with unit probability, the loss function l contains no reflection symmetry:
Theorem 1. Let E satisfy PropertyDandd. Assumption 7 Then, with probability 1 over the sampling of Oo, there exists no projection matrix and reflection point 0' such that & has the (",P)-yymmetry
The core mechanism of this theorem is decoupling the solutions of the symmetries from the solu- tions of the weight decay. With weight decay, solution with a small norm is favored, whereas with a random bias, the symmetric solutions are shifted by a small constant and no longer overlap with solutions that have a small norm. Equivalently, this method can be implemented as a random bias in the Ly regularization: f.(0, x) = E(O) + 7 + 0ol2.. This a result of the fact that simple translation does not change the nature of the function. There are two remarkable parts of the theorem: (1) it not only removes all existing symmetries but also guarantees that there are no remaining ones; (2) the proof works long as the variance of 6 is nonzero. This hints at the possibility of using small and random fg. which removes all symmetries in principle and also does not essentially affect the solutions of the original objective. In this work, we will refer the method in Eq. (6) as syre, an abbreviation for "symmetry removal."
5.2 UNCOUNTABLY MANY SYMMETRIES
In deep learning, it is possible for the model to simultaneously contain infinitely many reflection symmetries. This happens, for example, when the model parameters have the rotation symmetry or the double rotation symmetry (common in self-supervised learning problems Ziyin et al. (2023) or
An example that violates this assumption is when x(0,x) reflection symmetries everywhere and for every data point because is constant function.
oo 1(0,x)) oo has infinitely many
4

---

Published as a conference paper at ICLR 2025
transformers). It turns out that adding static bias and weight decay is not sufficient remove all symmetries, but we will show that a simple modification would suffice.
We propose to train the following alternative loss instead, where stands for "advanced moval":
(0) {(O 0o) 10
where D is a positive diagonal matrix in which all diagonal elements of D are different. The simplest way to achieve such D is to set Dis Uniform(1 e, +). where is small quantity.
Theorem 2. Any (0', P) -symmerry that aar satisfies obeys: (1) P0, 0' (2) and PD= DP.
Conditions (1) and (2) implies that there are at most finitely many (0', P)-ymmeerry far can have. When there does not exist any symmetry that satisfies this condition, we have removed all the sym- metries. In the worst case where é constant function, there are 2 symmetries where N is the number of reflection symmetries. If we further assume that every P is associated with at most finitely many 0', then we, again, remove all symmetries with probability 1. The easiest way to determine this D matrix is through sampling from uniform distribution with a variance ap << 1.
5.3 STRENGTH OF S SYMMETRY REMOVAL
While any level of 0g and dD are sufficient to remove the symmetries, one might want to quantify the degree which the symmetries are broken. This is especially relevant when the model is located close to symmetric solution and requires large gradient to escape from it. Also, related question that may arise in practice is how large one should choose dg and on, which are the variances of Oo and D. The following theorem gives quantitative characterization of the degree of symmetry breaking
Theorem 3. Let the original loss satisfy ())yyyyymmtrr where 0 Rd. Then, for any local minimum 9 e @(1), if = o(oo) and P0 + 0.
POO [e (0+ + 0-) -l (T 0*)] 2(700).
This theorem essentially shows a "super-Lipschitz" property of the difference between the loss func- tion values between parameters and their reflections. This means that with a random bias, the sym- metry will be quite strongly removed, as long as we ensure when aD As a corollary, it also shows that after applying a static bias, no symmetric solution where PO 0 can still be stationary point because as PO + O, the quantity A converges to the projection of the gradient onto symmetry breaking subspace.
< do, which is certainly ensured
Corollary 1. For any 0 such that PO = 0, e 2(yoo).
Now, the more advanced question is the case when there are multiple reflection symmetries, and one wants to significantly remove every one of them.
Theorem 4. Let contain N reflection symmetries: ((P;,0))) Let
A == (a + 07)
+
(9)
|P:0|
Then, for any local minimum 0 e (()), letting 7:00 2 for any e and 8 L.
2cN T-8
guarantees that Pr(min, [A,] > e) > d
In the theorem, the probability is taken over the random sampling of the static bias. Namely, the achievable strengths of -yrmetry- scales inversely linearly in N, the size of the minimal set of the entire group generated by N reflections. In general, without further assumptions there is no way to improve this scaling because, for example, the smallest of N independent bounded variables roughly scales as 1/N towards its lower boundary.
General Groups. Lastly, one can generalize the theory to prove that the proposed method removes symmetries from a generie group. Let G be the linear representation of generic finite group, possibly with many nontrivial subgroups. If the loss function & is invariant under transformation by the group G, then
Vge G, {(0) E(g0).
(10)
'Because Og and 1 are randomly sampled, the big-O notation O(2) is used to mean that as the scaling parameter tends infinity, 3oo such that Pr( co2)
5

---

Published as a conference paper at ICLR 2025
Because G' is finite, it follows that the representations g must be full-rank and unipotent.
The following theorem shows that at every symmetric solution, there exists an escape direction with strong gradient that pulls the parameters away from every subgroup of the related symmetry. In other words, it is no longer possible to be trapped in a symmetric solution. While general groups appear less common, they exist in lot of application scenarios with equivariant networks (Cohen & Welling 2016), where it is common to incorporate group structures into the model.
Letting U be subgroup of G, denote with the overbar the following matrix:
U
Σ u
(11)
U ueU
Note that U is a projection matrix: - = U. This means that --U U is also projection matrix. Importantly, U projects vector into the symmetric subspace. For any u e U. wU U. Likewise, 7 U projects any vector into the symmetry-broke subspace, well-known result in the theory of finite groups (Gorenstein] 2007). We denote by
Av = 1(1 V(v-
(12)
the strength of the symmetry removal for the subgroup V.
Theorem 5. Lei T(G) denote the smallest minimul generating set for the group G. Z denotes the number of minimal subgroups of G. Let i be invariant under the group transformation G and let be in the invariant subspace of subgroup U G. Then, for every subgroup V a U 4 G,
1. Av Q(yyorrank((I V)):
2. minyau Av (( (1
3. G is abelian, miny Ay
4. additionally, for any 0 and 6 < 1, Pr(miny Av > e) > 700 = Ω
(
1-8
Item (2) of the theorem is essentially due to the fact that removing symmetries from a larger group can be reduced to removing them from one of its subgroups. Conceptually, this result has the same root as the classical theory of combinatorial designs, where averag- ing over a subset of groups has the same effect as averaging over the whole group (Lindner & Rodger 2017). In general, 1 s 2 s |G] (and sometimes < [G], and so this scaling is not bad. For the part (3) of the theorem, the term [[((] is especially meaningful. It is well-known that [[U)] S log (U], and so the worst-case symmetry- breaking strength is only of order 1/ log (U), which is far slower than what one would expect. In fact, for finite group with size N, the number of subgroups can grow as fast Nilg (Borovik et al. 1996), and thus, one might naively think that the minimal breaking strength decreases as - log This theorem shows that the pro- posed method is highly effective at breaking the symmetries in the loss function or the model.
200
wd
syre
150
100
10a
102
10a
input dimension
Figure 2: With the proposed method, the rank of the feature covariance matrix increases with the input dimension. The same may happen under vanilla weight decay, but the effect is not strong. The covariance is computed from the first-layer output with batch size 1000, and eigenvalues larger than 10 are truncated to zero.
A numerical example is shown in Figure 2 which validates a major prediction of the theorem: symmetry easier remove if it is high-dimension two-layer ReL.U net in a teacher-student scenario and change the input dimension. This experiment holds the number of (permutation) symmetries fixed and directly controls rank(I-Wp -V.. As the input dimension increases, the symmetry of the learned model becomes lower. In comparison, without a static bias, having a high dimension is not so helpful.
We train
5.4 HYPERPARAMETER AND IMPLEMENTATION REMARK
As discussed, there are two ways to implement the method (Eq, 6 or Eq. 7 In our experiments, we stick the definition of Eq. (6). where the model parameters are biased, and weight decay is the same as the standard implementation. For the choice of hyperparameters, we always set a = 0
*This result can be generalized to the case where all projectors V commute with each other, even if G is nonabelian. Namely, this is consequence of the properties of the representations of G and of G itself.

---

Published as a conference paper at ICLR 2025
35
a- verilla +wd. +bies
0.935
vanilla
30-
W-fix
25-
0.930
150
syre
accuracy
20-
1 100
0.925
15
10-
0.920
syre
wd
0.95 1.00
1:05
10-1
10 0- 5000 10000 15000 20000
Figure 3: Training with syre in standard settings. The result shows that biasing the models by a small static bias does not change the performance of standard training settings. Left: Application of the method to linear regression problem. Here, a= d/N is the degree of parameterization. A well-known to prevent double descent when a= 1. Here. we see that the proposed method works as well as vanilla weight docay. Because there is no reflection symmetry in the problem. the proposed method should approximate vanilla weight decay. Mid: Test accuracy of Resnet18 18 on the CIFAR-10 datasets. The blue line denotes the performance of Resnet, and the shadowed area denotes its standard deviation estimated over 10 trials. For To 0.2, there is significant difference between the performance of the vanilla Resnet and syre Resnet. Right: linear regression with a redundant parametrization (Poon & Peyré 2021 J. The loss function takes the form E(u, w) (u @ w)" z v!?. Due symmetry, the point (us, wt) 0 is low-capacity state where the i-th neuron is "dead". Training with style, the model stayed away from any trapping low-capacity state during
of weight decay
training. In comparison, training with vanilla SGD or a heuristic for fixing the weights does not fix the problem of collapsing to a low-capacity state.
as we find only introducing dy to be sufficient for most tasks. Experiments with standard training settings (see the next section for the Resnet18 experiment on CIFAR-10) show that choosing do to be at least an order of magnitude smaller than the standard initialization scale (usually of order 1/vd d for a width of d) works the best. We thus recommend a default value of og to be 0.01/Vá, where 1/d is the common initialization variance. For the rest of the paper, state do in relative
units of vd for this reason. That being said, we stress that do is hyperparameter worth tuning, as it directly controls the tradeoff between optimization and symmetry removal.
6 EXPERIMENT
First, we show that the proposed method is compatible with standard training methods. We then ap- ply the method a few settings where symmetry known to be major problem in training. We see that removing symmetries with the proposed method is well correlated with improved model perfor- mance for these problems. Additional experiments are presented in Appendix [ including the time and memory usage of syre and its application to transformers and improving mode connectivity(" F
6.1 COMPATIBILITY WITH STANDARD TRAINING
Ridge linear regression. Let us first consider the classical problem of linear regression with d- dimensional data, where one wants to find min E, x, y;)². Here, the use of weight decay has well-known effect of preventing the divergence of generalization loss N = d (Krogh & Hertz 1992 Hastic et al. 2019). This is due to the fact that the Hessian matrix of the loss becomes singular exactly N d (at infinite N and d). The use of weight decay shifts all the eigenvalues of the Hessian by 7 and removes this singularity. In this case, one can show that the proposed method is essentially identical the simple ridge regression. The ridge solution is w" = =[ + A]- ".'[[yy) ry). where A [[[xx"] ". and the solution to the biased model is
critical dataset size
"º = E + (E[ry] + 700)
(13)
The difference is negligible with the original solution if either and Gg are small. See Figure left.
Reparametrized Linear Regression. A minimal model with emergent interest in the problem of compressing neural networks is the reparametrized version of linear regression (Poon & Peyre 2021 Ziyin & Wang 2023). the loss function of which is E(u, w) [ w) y|=, where we let u, w, I E I 2mm and y E R. Due to the rescaling symmetry between every parameter 2; and wis the solutions where ;; w; = 0 low-capacity state where the i-th neuron is "dead."
6 An implementation of syre can he found https///gttuuh.co///uxxxy//y//

---

Published as a conference paper at ICLR 2025
1000
dogroa of symmery
900
vertilla
W-fix
degrue symmetrr
800
out weight decey
700
syre
600
opt mai frontiar
850
500
800
2000
-1800
-1600
11400
-1200
1000
50
150
objective
objective
Figure 4: The degree of symmetry versus the objective value for two choices of B and various training methods with different hyperparameters. The proposed method the only method to smoothly interpolate between optimized solutions and solutions with low symmetry. syre performs well in both cases. Left: objective with unstructured symmetry Right: structured symmetry.
For this problem, we compare the training with standard SGD and syre. We also compare with heuristic method (W-fix), where a fraction 6 = 0.3 of weights of every layer is held fixed with a fixed variance heuristic for removing symmetries and is found to work well when there is permutation symmetry. We see that both the vanilla training and the W-fix collapse to low-capacity states during training, whereas the proposed method stayed away from them throughout. The reason is that the proposed method is model-independent and symmetry-agnostie working effectively for any type of possibly unknown symmetry in an arbitrary architecture.
0.01 at initialization. This method has been suggested in Lim al. (2024)
ResNet. We also benchmark the performance of the proposed method for ResNet18 with different do on the CIFAR-10 10 datasets in Figure When do 0, the syve model is equivalent to the vanilla model. Figure[sshows that the difference in performance between the vanilla Resnet and syre Resnet is very small and becomes neglectable when do 0.2.
6.2 BENCHMARKING S SYMMETRY REMOVAL
In this section, we benchmark the effect of symmetry control of the proposed method for two con- trolled experiments. To compare the influence of syre and other training methods on the degree of symmetry, we consider minimizing the following objective function:
(w (w=w)²² w" Bw := (w (w-wi w)
Σ ()F 1=1
where w Rd is the optimization parameter and B Raxd is given symmetric matrix with eigen- values X and eigenvectors Us (u,² 1). The objective function has n reflection symmetries Paw := w - 2(,," w)v, Hence, we define the degree of symmetry as Σ 1 [ w < c.h). where Oth is a given threshold. Depending on the spectrum of B, the nature of the task is different. We thus consider two types of spectra: (I) an unstructured spectrum where B = G + G for a Gaussian matrix G. and (2) a structured spectrum where B = diag(v) where v is a random Gaussian vec- tor. Conceptually, the first type is more similar to rotation and double rotation symmetries in neural networks where the basis can be arbitrary, while the second a good model for common discrete symmetries where the basis is often diagonal sparse. For the first case we choose Cth 10-3 and for the second case we choose 4th = 1
we compare syre, W-fix, drop out, weight decay, and the standard training methods in this setting for d = 1000 and two choices of B. In both cases, we use Gaussian initialization and gradient descent with a learning rate of 10-1 For syre and weight decay, we choose weight decay from 0.1 to 10. For W-fix, we choose from 0.001 to 0.1. For dropout, we choose a dropout rate from 0.01 to 0.6. Figure shows that for both cases, syre is the only method that effectively and smoothly interpolates between solutions with low symmetry and best optimization. This is strong piece of evidence that the proposed method can control the degree of symmetries in the model.
In Figure 4
6.3 FEATURE AND NEURON COLLAPSES IN SUPERVISED LEARNING
See Figure 5 where we train the vanilla and syre four-layer networks with various levels of weight decay 7 and input-ouiput covariance The dataset is constructed by rescaling the input by factor of a for the MNIST dataset. The theory predicts that the syre model can remove the permutation symmetry in the hidden layer. This is supported by subfigures in Figure 5 where vanilla training results in a low-rank solution. Meanwhile, the accuracy of the low-rank solution is significantly lower for large 7 or small a, which corresponds to the so-called neural collapses. Also,
8

---

Published as a conference paper at ICLR 2025
1.0
wd syre
wd
0.8
Syre
0.6
0.6
0.4
rani
0.4
0.2
0.2
10 10 10
10
Figure 5: Performance of 4-layer FCN for datasets with various weight decay 7 and data distributions with varyings strengths of linear correlation a. As the theory predicts, the covariance of the model with vanilla weight decay has low-rank structure and performs significantly worse. In the main figures, solid lines denote training accuracy and dashed lines denote test accuracy. The dashed black line corresponds random guess. Subtigures show the rank of the covariance matrix of the first layer output before (solid lines) and after (dashed lines) activation with batch size 1000. We set eigenvalues smaller than 10t to O. Left: a 1 and different 7 Right: y 0.01 and different A similar experiment for vision transformers is presented in Section B.5
Hyperparameter
Low- lowrrnkkness
Penult. Layer Acc.
Last Layer Acc.
vanilla syre
70%
46.8%
22.2%
oo = 0.1
0%
46.8%
31.7%
do = 0.01
0%
46.2%
32.5%
0g 0.1, all layers
0%
44.6%
30.7%
0 0.01, all layers
0%
45.4%
32.4%
Table I: Performance of the linearly evaluated & CIFAR 100 for the unsupervised self-constrastive learning task. Here, the low-rankness measures the proportion of eigenvalues smaller than 10 Our experi- ment indicates that symmetry-indiee induuced reduction model capacity can explain about 50% the performance difference between the representation of the two layers.
observe that syre shifts the eigenvalues of the representation by magnitude proportional to do. thus explaining the robustness of the method against collapses in the latent representation (See Figure
6.4 POSTERIOR COLLAPSE IN BAYESIAN LEARNING
Wang & Ziyin 2022) points out that type of posterior collapse in Bayesian learning (Lucas et al. 2019 Wang et al. )021) is caused by the low-rankness of the solutions. We show that training with sye could overcome this kind of posterior collapse. In Figure 9 we train a 3 V-VAE (Kingma
& Welling 2013 Higgins et al. 2016) on the Fashion MNIST dataset. Following Wang & Ziyin (2022), we use 3 to weigh the KL loss, which can be regarded as the strength of prior matching. Both the encoder and the decoder are a two-layer network with SiLU activation. The hidden dimension and the latent dimension are 200. Only the encoder has weight decay because the low-rank problem is caused by the encoder rather than the decoder. We also choose the prior variance of the latent variable to be 7jene 0.01. Other settings are the same as Wang & Ziyin| (2022). Posterior collapse happens at / = 10, signalized by large reconstruction loss in the right side of Figure 9 However, the reconstruction loss decreases, and the rank of the encoder output increases (according the left side of Figure 9 after we use weight decay and syre. This is further verified by the generated image
Figure 10 Therefore, we successfully remove the permutation symmetry of the encoder.
6.5 LOW-CAPACITY TRAP IN SELF-SUPERVISED LEARNING
A common but bizarre practice in self-supervised learning (SSL) is to throw away the last layer of the trained model and use the penultimate learning representation, which found to have much bet- ter expressiveness than the last layer representation. From the perspective of symmetry, this problem caused by the rotation symmetry of the last weight matrix in the SimCLR loss. We train a Resnet- 18 together with a two-layer projection head over the CIFAR- 100 dataset according to the setting for training SimCLR in Chen et al. (0200). Then, a linear classifier is trained using the learned rep- resentations. Our implementation reproduces the typical accuracy of SimCLR over the CIFAR-100 dataset (Patacchiola & Storkey 2020). As in Chen et al. (202)), the hidden layer before the pro- jection head is found to be better representation than the layer after. Therefore, method to the projection head or to all layers. According to Table syre removes the low-rankness of the learned features and increases the accuracy trained with the features after projection while not
apply our syre

---

Published as a conference paper at ICLR 2025
6000
75-
4000
raturn
50-
vanilla
vanilla
syre
syre
weight decay
weight decey
25 5M 10M
0M
15M
20M
25M
0M
SM 10M
15M
20M
25M
Figure 6: Loss of plasticity in continual learning in an RL setting. We use the PPO algorithm (Schulman et al. 2017) to solve the Slippery-Ant problem (Dohare et al, 2023). The rank and the performance of the vanilla PPO decrease quickly. while the rank and the performance of PPO with syre remain the same, beyond that of PPO with weight decay. Left: the effective rank of the policy network as defined in Dohare et al.) (2023). Right: returns. Each trajectory is averaged over different random sceds.
changing the representation before projection. Thus, symmetry-induced reduction in model capacity can explain about 50% of the performance difference between the representation of the two layers. Also, an interesting observation is that just improving the expressivity of the last layer is insufficient to close the gap between the performance of the last layer and the penultimate layer. This helps us gain new insight: symmetry is not the only reason why the last layer representation is defective.
6.6 LOSS OF PLASTICITY IN CONTINUAL LEARNING
A form of low-capacity collapse also happens during continual learning, i.e., the plasticity of the network gradually decreases as the model is trained on more and more tasks. This problem is common in both supervised and reinforcement learning settings and may also be relevant to the finetuning of large language models (McCloskey & Cohen. 1989 Ash & Adams 2020 Dohare al. 2021 Abbas et al. 2023).
Supervised Learning. In Figure we train CNN with two convolution layers (10 channels and 20 channels) and two fully connected layers (320 units and 50 units) over the MNIST datasets. For the data, randomly permute the pixels of the training and test sets for 9 times, forming 10 different tasks (including the original MNIST). We then train a vanilla CNN and a syre CNN over the 10 tasks continually with SGD and weight decay 0.01. The inset of Figure TT] shows that the rank of the original model gradually decreases, but the syre model remains close to full rank. Correspondingly, in the right side of Figure 11 the accuracy over the test set drops while the rank of the original model collapses, but the accuracy of the syre model remains similar.
Reinforcement Learning. In Figure 6 we use the PPO algorithm for the Slippery- Ant problem (Dohare et al. 2023), a continual variant of the Pybullet's Ant problem (Coumans & Bai 2016) with friction that changes every 5M steps. Hyperparameters for the PPO algorithm are borrowed from Dohare et al. (2023). and we use a weight decay of 0.002 for both PPO with weight decay and with syre. Figure 6 suggests that syre is effective in maintaining the rank of the model during continual training and obtains better performance than pure weight decay.
7 CONCLUSION
symmetry-inducee neural-network training problems exist extensively in machine learning. We have shown that the existence of symmetries in the model or loss function may severely limit the expressivity of the trained model. We then developed a theory that leverages the power of repre- sentation theory to show that adding random static biases to the model, along with weight decay, is sufficient remove almost all symmetries, explicit hidden. We have demonstrated the relevance of the method to a broad range of applications in deep learning, and a possible future direction is to deploy the method in large language models, which naturally contain many symmetries. Lastly, is worth noting that on its own, symmetry is neither good nor bad. For example, practitioners may be interested in introducing symmetries to the model architecture in order to control the capacity or hierarchy of the model (Ziyin et al.) 2025). However, with too much symmetry, the training of mod- els becomes slow and likely to contain many low-capacity traps. Meanwhile, a model completely without symmetry may have undesirably high capacity and be more prone to overfitting. Having the right degree of symmetry might thus be crucial for achieving both smooth optimization and good generalization. With our proposed method, it becomes increasingly possible to deliberately fine-grain engineer symmetries in the loss function, introducing desired symmetries and removing undesirable ones.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C. Machado, Loss of plasticity in continual deep reinforcement learning. arXiv preprint arXiv:2303.07507, 2023.
Jordan Ash and Ryan P Adams. On warm-starting neural network training. Advances in neural information processing systems, 33:3884-3883 2020.
Alexandre Borovik, Laszlo Pyber, and Aner Shalev. Maximal subgroups in finite and profinite
groups. Transactions of the American Mathematical Society, 348(9):3745-3761, 1996.
Marco Cariglia. Hidden symmetries of dynamics in classical and quantum physics. Reviews of Modern Physics, 86(4):1283, 2014.
Feng Chen, Daniel Kunin, Atsushi Yamamura, and Surya Ganguli. Stochastic collapse: How gradi- ent noise attracts sgd dynamies towards simpler subnetworks. arXiv preprint arXiv:230600425,, 2023.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. arXiv preprint arXiv: 18/2.0996, 2018.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International confer- on machine learning, pp. 2999. PMLR, 2016.
Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning, 2016.
Dinh, R. Pascanu, Bengio, and Y. Bengio. Sharp Minima Can Generalize For Deep Nets. ArXiv e-prints, March 2017.
Shibhansh Dohare, Richard Sutton, and A Rupam Mahmood. Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint arXiv:2108.06322, 2021.
Shibhansh Dohare, Fernando Hemandez- Garcia, Parash Rahman, Richard S Sutton, and A Rupam Mahmood. Maintaining plasticity in deep continual learning. arXiv preprint arXiv:2306.1334
2023.
Rahim Entezari, Hanie Sedghi, Olga Saukh. and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. arXiv preprint 10,0269, 2021.
Valeri P Frolov, Pavel Krtouš, and David Kubizñák. Black holes, hidden symmetries, and complete integrability. Living reviews in relativity, 20:1-221, 2017.
Kenji Fukumizu. A regularity condition of the information matrix of multilayer perceptron net- work. Neural networks, 9(5):871-87, 1996.
Charles Godfrey, Davis Brown, Tegan Emerson, and Henry Kvinge. On the symmetries of deep learning models and their internal representations. Advances in Neural Information Processing Systems, 35:1 15931199119955 2022.
Ian Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empiri- cal investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211,222 2013.
Daniel Gorenstein. Finite groups, volume 301. American Mathematical Soc., 2007.
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation. arXiv preprint arXiv: (a0ee::03300800 2019.
11

---

Published as a conference paper at ICLR 2025
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. 2016.
Diederik Kingma and Max Welling, Auto-encoding variational arXiv: 14, 2013.
bayes.
arXiv preprint
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska BaraaBsaain,, et al. Overcom- ing catastrophic forgetting neural networks. Proceedings of the national academy of sciences, (4((13)3551-3524 2017.
Seijin Kobayashi, Yassir Akram, and Johannes Von Oswald. Weight decay induces low-rank atten- tion layers. arXiv preprint arXiv:2410.23819, 2024.
Anders Krogh and John Hertz. A simple weight decay improve generalization. In Advances in neural information processing systems, pp. 950-957, 1992.
Xingguo Li, Junwei Lu, Raman Arora, Jarvis Haupt, Han Liu, Zhaoran Wang, and Tuo Zhao. Sym- metry, saddle points, and global optimization landscape of nonconvex matrix factorization. JEEE Transactions Information Theory, 65(6))3489-33) -514,, 2019.
Derek Lim, Moe Putterman, Robin Walters, Haggai Maron, and Stefanie Jegelka. The empirical impact of neural parameter symmetries, or lack thereof. arXiv preprint arXiv:2405.20231 2024.
Charles C Lindner and Christopher Rodger. Design theory. Chapman and Hall/CRC, 2017.
James Lucas, George Tucker, Roger Grosse, and Mohammad Norouzi. Don't blame the elbo! a
linear vae perspective on posterior collapse, 2019.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization deep learning. arXiv preprint arXiv:1412.661/ 2014.
Massimiliano Patacchiola and Amos J Storkey. Self-supervised relational reasoning for representa- tion learning. Advances Neural Information Processing Systems, 33:4003-4014, 2020.
Clarice Poon and Gabriel Peyré. Smooth bilevel programming for sparse regularization. Advances in Neural Information Processing Systems, 34:1543-1555, 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
Berfin Simsek, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerst- ner, and Johanni Brea. Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances. In International Conference on Machine Learning, pp. 9722-9732. PMLR, 2021.
Ryan Tibshirani. Equivalences between sparse models and neural networks. Working Notes. URL https:///ww. stat. cmu. edu/" rrantiblhhaperssppasityynn pdf. 2021.
Yixin Wang. David Blei, and John Cunningham. Posterior collapse and latent variable non- identifiability. Advances in neural information processing systems, 34:5443-5455, 2021.
Zihao Wang and Liu Ziyin. Posterior collapse of linear latent variable model. Advances in Neural
Information Processing Systems, 35:37537-37548, 2022.
Kentaro Yoshioka. and related models on cifar-10
rision-transformers-cffaarrrr https: ptteps://qithnn com/ hemm/kketarooy777
Training vision transformers (vit)
vision-trans cow.ntrrrannoomrsssiiiiiii
2024.
12

---

Published as a conference paper at ICLR 2025
Bo Zhao, Robert M Gower, Robin Walters, and Rose Yu. Improving convergence and generalization using parameter symmetries. arXiv preprint arXiv:230S. 13404, 2023.
Liu Ziyin. Symmetry induces structure and constraint of learning. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview net / foormmide
7AFOAMI 4AE
Liu Ziyin and Zihao Wang. spred: Solving L1 Penalty with SGD. In International Conference on Machine Learning. 2023.
Liu Ziyin, Ekdeep Singh Lubana, Masahito Ueda, and Hidenori Tanaka. What shapes the loss landscape of self supervised learning? In The Eleventh International Conference Learning Representations, 2023. URL
Liu Ziyin, Mingze Wang, Hongchao Li, and Lei Wu. Parameter symmetry and noise equilibrium of stochastic gradient descent. In The Tiihth- Annual Conference on Neural Information Processing Systems, 2024.
Liu Ziyin, Yizhou Xu, Tomaso Poggio, and Isaac Chuang. Parameter symmetry breaking and restoration determines the hierarchical learning in ai systems. arXiv preprint arXiv:2502.05300, 2025.
13

---

GD converges to Published as a conference paper at ICLR 2025
A THEORETICAL CONCERNS
A.1 PROOF OF PROPOSITION I
Proof. Part (I). Note that have
0" = Pd (II = -II (I-P)8
(14)
Thus,
(I (I-2P) P)0 = -1I -I--P)0'
(15)
Therefore, have
+)), (16)
This proves part (1). Part (2). By definition,
f'(0 - ot +8) ) - f(0+0')
(17)
=[((I=P)0++H 0')
(18)
= (((((IP)) 8))
(19)
This completes the proof.
A.2 PROOF OF PROPOSITION 2
Proof. By 2 (z, 8) simplifies kernel model
g(x, 0) (x)
(20)
Let us consider the squared loss {(0) Σx (11l g(x, 0)12 and denote E.
P) e fxx Vv, fe, (xx,0))(I P), b (1 P.E, fe,f 0o) yzx. Assuming the learning
rate to be the GD reads
0²+1 0¹ 27(( A0' b),
(21)
where 8 If
11 22mar A))
(22)
0r lim
E1
(1) k 2mb
(-000 k=0
(23)
A b.
which is the well-known least square solution.
A.3 PROOF OF PROPOSITION
Proof. According to Ziyin 2024 Theorem 4), we have
PVol(x,0)) 0.
(24)
Therefore, after one step of GD or SGD, we still have P01 0. By induction, we have P0, - 0.
Finally, suppose that [a ddrank((P) forms basis of kerP, and define f' 0') ==
f(x, Ed dd-rank(P) f(a;) for dim = d rank(P). By choosing 8, 8T a; we have f(z 8.)
fxx,0)
14

---

Published as a conference paper at ICLR 2025
A.4 LEMMAS
Lemma 1. Let I E IKd and P be a projection matrix. Let (3) be scalar function that satisfies
f(x+x') f((1 (()11))) x') c Px,
(25)
where e is a constant vector. Then, there exists a unique function g(x) such that
1. g(z) has the x' c'cenricc P-symmetry,
2. and [(x) x()) 2 Px.
Proof. (a) Existence. f(x) g(x) + 10. x. Let us suppose g(x) is not c'-ceriic P-symmetry. Then, there exists r such that
(((e))))(((2)+))))0300 = +
(26)
Then, by definition, we have that
Px - [(12+)++
(27)
= =((x+x'-- 1 T
( ((I(-)))+t))
(28)
2
A+ 0 P(
T 00 P([ 2P) +')
(29)
+ o Px.
(30)
This is a contradiction. Therefore, there must exist g(x) that satisfies the lemma statement. (b) Uniqueness. Simply note that the expression of is uniquely given by
= f(x
(31)
A.5 PROOF OF THEOREM
Proof. We prove by contradiction. Let suppose there exists such pair, (O'P). By definition,
have that
f. 0') E(O 0' 0o) 0
(32)
By assumption, we have that
((I-2P)) + 0' + 0o) -1 0'1 1(0+ 0' 0o) + 7/10 0'13,
(33)
and, so, for all 0,
(((I2PP) + + +00 = ((0 + 06) + 4-00 AT P8' There are two cases: (1) P0' and (2) P0' O.
(34)
For case (1), we have that f((1 2P)0 + 0' + 0o) = E(O + 0' + 0o), but this can only happen if the original loss l has the (8' 0g) -enttrc P P-symmetry. By Property [ this implies that
8 + 0o o
(35)
for some Applying Pon on both sides, we obtain that
Pd. = P0.
(36)
But, Og is random variable with full-rank covariance while the set (PO.) has measure zero in the real space, and so this equality holds with probability zero.
Alternatively, note that et is odd and that ((x) can be shifted by constant to be an even function. The uniqueness follows directly from the fact that every function be uniquely factorized into an odd function and function.
15

---

Published as a conference paper at ICLR 2025
For case (2), it follows from Lemma that for fixed æ and 80. ((0) can be uniquely decomposed in the following form
= (0)==(())) 270" PO'
(37)
where 5(0) has the 0' + On-centric P-symmetry.
Let co P'' and 0 = P(O' + 0o). Then E(0)+ cg0 has the O-centric P symmetry. We also have θ- 2y P8p. According Assumption there are only countable many such {co, ff pairs. However,
P0, is standard Gaussian random variable, which leads to a contradiction with probability 1. O
Remark. It is easy to see that Assumption could be slightly relaxed. We only require {0 have a zero measure, which is also a necessary and sufficient condition.
ca to 2
A.6 PROOF OF THEOREM!
Proof. We prove by contradiction. Let suppose there exists such pair, (0',P). By definition, have that
(ar 0') 6(0 0' 0o) 7/10 0'1..
(38)
By assumption, we have that
8((I- + 111 L(0+ 0o) + + 0'1
(39)
and, so, for all 0,
E((I 2P) 0 06) 00' 0b) -0t PD((I (P00((P))+0000 8¹).
(40)
There are two cases: (1) PD((I- P)0 0') 0 and (2) PD((I 0') +0.
Like before, there are two cases. For case (2), the proof is identical, and so we omit it. For case (1), it must be the case that for some P. and 0'
- This is possible if and only if P(0'+ +0a) 0 for some i and P P. for the corresponding projection matrix. However, because P0' 0, this requires that
6' + 0o) = E(O 0' + 6o).
(41)
P(d' + 00) 6¹. By the definition of the reflection symmetry, we have that
(42)
Pg' 8'.
(43)
This means that At the same time, we have which implies that
0' 0 Pfo-
(44)
PD((I P)0 + =0,
(45)
PD(I p) -PD'
(46)
Because the right hand side is a constant that only depends on 0. This can only happen if both sides are zero, which is achieved if:
PD(I - =0
(47)
and
0: = PPo
(48)
The first condition implies that
PD PDP DJ DP,
(49)
which implies that and D must share the eigenvectors because they commute. Noting that
po =
(50)
we obtain that 0f is an eigenvector of P and so o is an eigenvector of D, but D is diagonal and with nonidentical diagonal entries, 8t much then be a one-hot vector, and P must also be diagonal and consists of values of I and 0 in the diagonal entries.
16

---

Published as a conference paper at ICLR 2025
A.7 PROOF OF THEOREM
3
Proof. First of all, note that shifting parameters of & is the same as shifting the parameters of the weight decay. Therefore, for any local minimum 0'
7 + Lac (0)) +
0o13
(51)
where 0 = 0g.
Therefore,
(a' + +0") (aT + +0*)
(52)
((0+07) + +1 1-0 -bol: ([11-2P)0+0) - - -P)0 0- -0000+0000000000000000 -0o1
(53)
-110++0-0003 - - 0" -0lli
(54)
+ -D + D(0-
(55)
where we have used the definition of reflection symmetry the third line. the fourth line, we have defined 21 = PO and 2 (I P)O. Thus,
101 = |=1l² |=1ll = 0(1)
(56)
Thus, we have that
(lar 0²) (ar((I- + 0)))
O D(0" -6))
(57)
=0((ol =1 [[)o0l==11,
(58)
where we have used the fact that each element of 0 - 0n is (20)) because 0" is arbitrary constant and 0n NN,0 2 By the assumption that on o(an). we obtain the desired relation
Ear (O' 0²) ( + 0"
[27700)
(59)
This finishes the proof.
A.8 PROOF OF THEOREM 4
Proof. First of all,
Pr(min|A./>/ Pr(|A1 > cA. >CA...//N/2))
(60)
maxx(1, (0, EP((A))))- -
(61)
where we have applied the Frechet inequality in the second line.
The sum Σ; Pr(|Ad> e) can also be lower bounded if each A is Gaussian variable with variance
N Pr(IA > E) 2 E1
N
2e
Σ
(62)
2
2cNV
2 N
(63)
min; V2TO
Thus, for Pr(ming |A, e) to be larger than 1 we must have
2cN
min oi 2 V2T(1-
(64)
Now, we show that A,-s are indeed Gaussian variables. From the previous proof, it is easy to see that for unit vector n,,
A A -6) o(70o)
(65)
Therefore, A; is Gaussian variable with standard deviation yop- Thus, min; a, 700- Thus, have obtained the desired result
2cN
700 2 V2T(1 - 8)
(66)
17

---

Published as a conference paper at ICLR 2025
A.9 PROOF OF THEOREM!
5
Before we start proving Theorem 5 we introduce a definition that facilitates the proof.
Definition 2. (Symmetry reduction.) We say that removing symmetry from group G reduces to removing the symmetry due to group G2 if for any vector
(1-G2)ml s [(1 Gi)n|
(67)
Now, we are ready to prove the main theorem.
Proof.
We first show (I - V)" Vol(O)
( = 0. For any é V and a R, we have {(0 2m) = [(((0 zn)),
(68)
where n is an arbitrary unit vector. Taking the derivative with respect to z, and recalling that g0 we have
0,
= =
(69)
Accordingly, we have
(Vn) VV
(gn) n² vo((O.
(70)
gEV
Due to the arbitrary choice of n, we have (1 - v) =0.
Therefore,
(I- (I-V)" volar(t) - -1- (1- -o01
(71)
27 0o)
(72)
-)) + 00700 toD))
(73)
As Oo is Gaussian vector with mean 0 and variance oZ. |1(1 - is Gaussian vari- able with mean 0 and variance || - VI = Q[rank(I- - V)oo), which gives |(I - V)
M(roo rank(T V)).
Now, we prove part (2) of the theorem. Note that if V a U and if 6g e ker V, then d e ker k. This means that for any group U such that V U and any vector do
(I-V)6l < 10l-
(74)
This means that to remove the symmetry from a large group U, it suffices to remove the symmetry from one of its minimal subgroups. Thus, let MG denote the set of minimal subgroups of the group G, have
min |(I - V)ool 2 min (I-V))ol) V4G
(75)
VaMa
The number of minimal subgroups strictly upper bounded by the number of elements of the group because all minimal subgroups are only trivially intersect each other. This follows from the fact that the intersection of groups must be a subgroup, which can only be the identity for two different minimal subgroups. Therefore, the number of minimal subgroups cannot exceed the number of elements of the group. This finishes the second part of the theorem.
For the third part, we show that the symmetry broken subspace of any subgroup contains the symme- try broken subspace of group generated by one the generators and it suffices to only remove the symmetries due to the subgroups generated by each generator. Let introduce the following notation for a matrix representation = of group element:
1 ord(s)
Σ
(76)
(rd
where ord(2) denotes the order of z. This equivalent to the symmetry projection matrix of the subgroup generated by z.
18

---

Published as a conference paper at ICLR 2025
Now, let G' be abelian. Then, both U and V are abelian. Let denote by I(U) {2+ the mininal generating set of U. Suppose that for all # 0 such that n im (V), must have é im 2T) for all j. This means that
& Uim(T 2z).
(77)
Or, equivalently,
n iz)
(78)
However, the space N (n)m(z7) S im (m because V a subgroup of U, which is generated by "1,", Am- To see this, let E im(zz) for all then,
(79)
for all j. Now, let II. dd(o)
V, we have
(I-V)n - (I (1-EII a(0))
21 n
(80)
This means that is in both iV) and Therefore, as long as T-V is not rank 0, it must share a common subspace with one of the 7- 2j, and so removing the symmetry from any subgroup of U can be reduced to removing the symmetry from the cyclic group generated by one of its generators from the minimal generating set S
which possible only if = 0 contradiction.
Therefore, we have proved that removing symmetries due to any subgroup of U can be reduced to removing the symmetry from (proper or trivial) subgroup of each of the cyclic decompositions of the group U, each of which is generated by a minimal generator of U. By the fundamental theorem of finite abelian groups, each of these groups is of order p for of these groups is cyclic, it contains exactly nontrivial subgroups. Taken together, this means that Am, we only have to remove symmetries from at most if [U| P) ...Pmm,
prime number p. Because each
m Σ
kj log U
(81)
many subgroups. This completes part (3).
For part (4), we denote A; (1- V for i I,('), According to (73). A, is approximately Gaussian variable with zero mean and variance rrannk (T V) Therefore,
IT)) Σ P((A))=)221((U))
2e(I((U)
(82)
min; 2 rI-
For Pr(min: [A e) to be larger than 1- 6, we must have
[e) V 2T -I (1
700
(83)
"This holds even if V is subset of (2z).
19

---

Published as a conference paper at ICLR 2025
300
250
275
250
150
wd
200
wd
syre
50
175
0.0
0.2
0.4
0.6
0.8
0.0
0.5
1.0
1.5
input noise
Figure 7: For same setting Figure the rank of the covariance matrix decreases with the label noise and increases with the input noise. We set eigenvalues smaller than 10 to zero.
10
10a
Gensioy
10a
densily
10-1
10
10-2
10
10-1
10-2 egenvalue
10a
10a
10-1
10-2
10
10a
eigenvelue
Figure 8: The spectrum of the covariance matrix of the model with vanilla weight decay (Left) and the syve 1. Clearly, the vanilla model learns low-rank solution.
model (Right) for y 0.01 and
B
ADDITIONAL EXPERIMENTS AND EXPERIMENTAL DETAIL
B.1
TREACHRRSSTUENN SCENARIO
This section gives some additional details and additional experiments in the teacher-student scenario Figure 2 Specifically, we implement a two-layer network with tanh activation, 300 hidden units, and different input units. The network outputs ten-dimensional vector corresponding to ten dif- ferent classes. We then randomly generate such network the teacher model, 10000 standard Gaussian samples the training set, and 1000 standard Gaussian samples. For both the syve and the vanilla model, we choose the Adam optimizer, learning rate 0.01, and weight decay 0.01.
As additional experiments, we also measure the influence of noisy labels and noisy input on the rank of the model in Figure 7 For the label noise, we randomly change 0% 80% of the labels, and for the input noise, add Gaussian noise to the input with standard deviation 0 to 1.6. Figure 7 suggests that the rank of the vanilla model decreases in the face of noisy labels and increases in the face of noisy input, perhaps because the latter can be regarded as data augmentation. The syre model, however, is not affected.
B.2 SUPERVISED LEARNING
This section presents some additional experiments for Section 6.3 Figure X gives the eigenvalue distribution of the networks in Fig which further supports the claim that the vanilla network leads to low-rank solution. In all the experiments above and in Section 6.3 we use four-layer FCN with 300 neurons in each layer trained on the MNIST dataset with batch size 64.
20

---

Published as a conference paper at ICLR 2025
200
35
p syre == syre
150
reconstructionlos los
==
rank
100
25
20
50
15
10
10 10"²
100
10
Figure 9: Rank and reconstruction loss for a VAE is trained on the Fashion MNIST datasct. The covariance of the model with vamilla weight decay has low-rank structure and larger reconstruction loss. More importantly, posterior collapse happens 3 but mitigated with weight decay. Left: the rank of the encoder output with batch size 1000. We set eigenvalues smaller than 10"H to O. Right: reconstruction loss vanilla and syre models.
Figure 10: Examples of Fashion MNIST reconstruction with syre and / 10. Left: No weight decay. Right: 1000. Clearly, the posterior collapse is mitigated by imposing syre with weight decay.
00
Tenk
Figure 11: Performance and accuracy of CNN trained continual Icarning task (permuted MNIST (Goodfellow et al. 2013 Kirkpatrick al. 2017)). The main figure shows the test accuracy, and the subfigure shows the rank of the convolution layers output with batch size 1000, where we set eigenvalues smaller than 10 4 to O. The results suggest that the rank of the model with vanilla weight decay gradually decreases, and the model completely collapses after the sixth task, while the syre model remains unaffected.
B.3 POSTERIOR COLLAPSE
See Figures 9 and 10
B.4 CONTINUAL LEARNING
See Figure
21

---

Published as a conference paper at ICLR 2025
0.8
wd
wd
syre
Syre
W-asym
W-asym
0 ssym
a-asym
0.4
0.4
0.2
10
10
10
10
Figure 12: Test accuracy and rank of VIT for CIFAR1O (adapted from Yoshioka (2024) with various weight decay y and data distributions with varyings strengths of linear correlation As the theory predicts, the covariance of features of the model with vanilla weight decay has a low-rank structure and performs significantly worse. Dashed black lines correspond to random guesses. Subfigures show the rank of the covariance matrix of the final layer input with batch size 512. We set eigenval- ues smaller than 10- Left: and different y. Right: 0.0001 and different a.
batchsize
MLP
syre-MLP
ResNet
syre-ResNct
ViT
syre- ViT
512
95 0.1
97
18.4 0.7
19.2 0.9
80 1
84
256
48 + 1
49
18.6 0.2
19.6 + 0.4
+ 5
57
128
26 1
2
19.9 0.4
20.9 0.3
39 1
47
Table 2: The per-batch time (ms) of various models.
batchsize
MLP
syre-MLP
ResNet
syre-ResNet
VIT
sye- ViT
512
382
384
7020
7049
4216
4218
256
360
382
3936
4254
2482
2484
128
360
382
2096
2190
1574
1574
Table 3: The GPU memory usage (M) of various models.
B.5 VISION TRANSFORMER (VIT)
Similar to Section 6.3 we train the vanilla and syre ViT with various levels of weight decay y and various levels of input-output covariance The dataset is constructed by rescaling the input by factor of a for the CIFARIO dataset. As the theoretical prediction, we can see that syre removes the rotation symmetries symmetry of transformers (Kobayashi et al. (024) and leads full-rank solution with higher accuracy, as shown in Figure 12 Moreover, syre also performs better than the W-asymmetric and o-asymmetric networks proposed in Lim al. (2024). We implement or- asymmetric networks using the recommended hyperparameters in Lim et al. (2024), and implement W-asymmetric networks by setting the same hyperparameters for each layer and doing a grid search.
B.6 TIME AND MEMORY CONSUMPTION
The time and memory consumption of various models used in previous sections is given in Tables ( In all experiments, we train the models on the CIFAR10 dataset with a signal A6000 GPU. clear that syre has neglectable influence time and memory consumption. This is expected because syre only adds static bias to training.
B.7 LINEAR MODEL CONNECTIVITY
We also test the influence of syre on the linear model connectivity with the same setting as Lim et al. (2024). Specifically, we obtain two well-behaved MLP with parameters 0, and 02, and test the performance of another MLP with parameters A01 + (1 2)0, for the permutation symmetry of MLP, expect that the MLP with parameters A01 + (1 2)0z also performs well. Figure suggests that syre performs similarly to W-asymmetric networks proposed in Lim et al. 2024), while methods have much fewer hyperparameters.
< 1. By removing
22

---

Published as a conference paper at ICLR 2025
syre
0.4
MLP
0.930
test loss
0.3
W-asym
o-asym
0.925
test
0.900
0.875
W
0-azym
0.1
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
interpolation
interpolation
Figure 13: Linear mode connectivity: test accuracy and loss curves along linear interpolations be- tween trained networks. We train MLPs over the MNIST dataset.
1250
gradient
1250
gradient uppeet bound
upperr bound
300
300
250
250
0
20
40
20
Fxxe points
fixed points
Figure 14: Ranks of the gradient covariance and NTK (Left: beginning of training. Right: end of training.) against the theoretical upper bound implied by Propositions 3 and 2 Two observations agree with expectation: (1) the ranks of both matrices decrease more and more neurons stuck at the entrapped subspace, and (2) the ranks are always upper bound by the theoretical upper bound.
C SYMMETRY AND RANK OF THE NTK
In this section, we show that the rank of the NTK and gradient reduces as more and more of the neu- rons in the symmetric state (namely, the fixed point induced by the group averaging projector).
Due to the difficulty in computing the spectrum the NTK, we restrict to a two-layer subnetwork within large network trained MNIST with the number of neurons 10 64 10, with 1280 many parameters. This means that the gradient second moment has dimension 1280 1280, and the model is trained 1500 data points. Thus, the empirical NTK be dimension 1500 1500. Here, the symmetry projection we consider due to the permutation symmetry, whose projection has a rank of 20 (because every neuron has outgoing weights and incoming weights). Thus, according to the Propositions 3 and 2 the maximum possible rank for the NTK and the gradient is 1280 20 x n, where is the number of neurons projected to the symmetry fixed point. See Figure
matrix having
23


Reviews:
[{"review_id": "KjhJ9M4T0W", "reviewer": "Reviewer_Fj68", "summary": "This paper presents a novel approach for addressing the limitations posed by symmetries in neural network training, which often lead to low-capacity model states. The authors propose a method called \"syre\", which can effectively remove almost all types of symmetries from neural networks, thereby enhancing model expressivity and performance across various training scenarios. The method is both model and symmetry group agnostic and straightforward to implement, requiring only minimal changes to existing training pipelines.", "questions": "1. It would be beneficial to see further exploration into the limits of the method, especially in more complex or less controlled environments than those tested. Also, maybe trying some transformer architecture based methods for SSL might make the architecture agnostic point stronger? \n\n2. Could the authors elaborate on any potential computational overhead or scalability issues when implementing syre in larger or more complex models?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is theoretically sound. Authors have rigorously proved many useful results like Proposition 3 which shows symmetry reduces parameter dimension.\n2. Their proposed solution is generalizable across different architectures and does not require detailed knowledge of the model’s symmetries.\n3. The empirical evaluations demonstrate significant improvements in model performance and capacity, particularly in challenging scenarios like continual learning and self-supervised learning setups.", "weaknesses": "1. Some of the theoretical explanations are dense and might be challenging for readers not familiar with the detailed aspects of symmetry in neural networks. Having decent knowledge of symmetries in neural networks I still struggled a bit with some sections like Section 5. Summarizing early on what the main theoretical result of each section might make it easier for reader to appreciate the contributions of the paper better. \n\n2. The limitations of the method are not thoroughly discussed, which could be beneficial for practitioners to know when to use this technique with caution. \n\n3. The formatting of the paper seems off. All the headings are pretty close to the previous paragraph hurting the readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BSElNowAOZ", "reviewer": "Reviewer_JSoA", "summary": "Main contributions are:\n- Builds on (Ziyin 2024) to introduce two ways that symmetry may reduce model capacity— a) that in the lazy training limit, the P-symmetry ‘masks’ certain features around a symmetric solution and b) the parameter dimension is reduced by the rank of P when around a symmetric solution. \n- Introduces a straightforward approach to eliminate these symmetric solutions by both fixing a random bias on the neurons (to handle finite symmetries) and also modifying weight decay to use a norm with respect to a positive diagonal matrix D (to handle infinite symmetries, e.g. like those that appear in the residual stream of a transformer). Also introduce a measure to quantify the degree to which a symmetry is broken. \n- Experiments that measure the extent of symmetry removal for some linear regression set-ups and some vision tasks, and continual learning in an RL setting.", "questions": "- How does your syre do on the (Lim 2024) LMC experiments?\n- Do you have any (even at a very high level) design rules for how a training-algorithm designer may usefully modulate the symmetries in a model?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- As noted, provide some plausible and interesting explanations for some mechanisms for how symmetry may reduce capacity that nontrivially build on (Zivin 2024).\n- In some settings and experiments, their algorithm (syre) seems to usefully outperform baselines/another symmetry removal method?", "weaknesses": "Experimental shortcomings\n- The authors make strong claims around the ability of symmetry-removal to improve performance for a) SSL, b) a supervised continual learning problem, and c) an RL continual learning problem. They do not implement proper baseline comparisons for any of these settings nor go beyond toy experiments. If they want to claim that (some degree of) symmetry removal is helpful for training models in practical settings, they need to either show that a) symmetry removal is reasonably competitive in mildly (compute/FLOP-controlled or at least aware) realistic settings or b) symmetry removal explains some % of the success of methods that do well in these settings. Aside from performance gains, benchmarking their method against the (Lim 2024) methods on linear mode connectivity would also help better validate their set-up. Likewise, benchmarking on other approaches on symmetry-removal.\n- On the other hand, the authors only very loosely test their explanations in Prop. 2/3. Presumably, if this explanation is correct, shouldn’t models with more/less symmetries be less/more stuck in neuron-collapsed solutions (e.g. in the spirit of the Zivin 2024 experiments)? Fig. 1 only shows one case. If this claim around the science of models is to be taken as a main contribution of the paper, more experimental evidence of the propositions is needed.\n\n\nMiscellaneous\n- Typo on line 65-66: “$\\|\\theta\\|_D^2:=w^T D w$ is the norm of $w$ with respect to $D$” should be “$|w|_D^2:=w^T D w$ is the norm of $w$ with respect to $D”\n\n\nZiyin, Liu. \"Symmetry Induces Structure and Constraint of Learning.\" Forty-first International Conference on Machine Learning.\nLim, Derek, et al. \"The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof.\" arXiv preprint arXiv:2405.20231 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PuIwZiEflq", "reviewer": "Reviewer_PYEZ", "summary": "This paper addresses the problem of symmetries in neural network loss landscapes and their role in causing models to get trapped in low-capacity states. The authors first prove that symmetries in the loss function can lead to reduced model capacity through two mechanisms: feature masking and parameter dimension reduction. They then propose SYRE, a simple modification to neural network training that aims to remove symmetries by adding a random bias to the parameters along with weight decay. For cases with uncountably many symmetries, they propose an advanced version that additionally uses a random diagonal matrix in the weight decay term. The method is model-agnostic and requires no knowledge of the underlying symmetries. The authors demonstrate their method's effectiveness across several applications where symmetry-induced capacity reduction is a concern, including supervised learning, variational autoencoders, self-supervised learning, and continual learning. They particularly show improvements in scenarios where models typically suffer from various forms of collapse, such as posterior collapse in VAEs and loss of plasticity in continual learning.", "questions": "Those are good questions. Here are those formatted plus a few additional ones I think would be helpful:\n\n1. **Fixed Points vs Symmetries**:\n   - Do you mean fixed points (invariant sets) when discussing \"symmetric points\" where $P\\theta_0 = 0$? If so, consider revising terminology to avoid confusion with symmetry orbits.\n   - Why focus on fixed points rather than the full symmetry orbits which characterize the degeneracies in the loss landscape?\n\n2. **Gradient Analysis**:\n   - Can you clarify the argument around equation (2) about vanishing gradients? How does this differ from the natural vanishing of gradients along symmetry orbits?\n   - In the expansion around fixed points, do you consider gradients in directions orthogonal to the symmetry?\n\n3. **Initialization and Fixed Points**:\n   - For a simple network $F(x) = UVx$, the origin $(U,V)=(0,0)$ is a fixed point, but why is this relevant when standard initialization (Glorot, etc.) avoids such points?\n   - Do your arguments about trapping at fixed points apply to more realistic initialization schemes?\n\n4. **Regularization Effects**:\n   - Can you provide ablation studies comparing your method with standard weight decay?\n   - For the D-norm modification, how does making weight decay ellipsoidal fundamentally differ from standard weight decay in terms of symmetry breaking?\n\n5. **Theoretical Claims**:\n   - The condition $P\\theta' = \\theta'$ seems unnecessarily restrictive. Can your results be generalized without this condition?\n   - For continuous symmetries (like $GL(h)$ in linear networks), can you make precise what subset of symmetries your method breaks?\n\n6. **Empirical Validation**:\n   - Could you provide standard weight decay baselines for all experiments?\n   - In scenarios where your method outperforms standard weight decay (like continual learning), what mechanism explains this improvement?\n\nWould you like to add any other questions?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Here are the main strengths of the paper:\n\n1. Addresses an important practical problem of symmetry-induced model collapse that affects multiple domains in deep learning (VAEs, continual learning, self-supervised learning).\n\n2. Proposes a remarkably simple solution that requires minimal code changes and no architectural modifications, making it easily adoptable in practice.\n\n3. Shows promising results in continual learning scenarios, demonstrating improved plasticity and performance maintenance over time compared to standard approaches.\n\n4. Makes an interesting connection between symmetries and model collapse, providing a framework for thinking about how symmetries might limit model expressivity.", "weaknesses": "1. **Conceptual confusion about symmetries**: \n   - The paper discusses \"removing symmetries\" but primarily analyzes fixed points where $P\\theta_0 = 0$\n   - This misses the key aspect of symmetries - their group orbits and the associated degeneracies\n   - In the simple case of a two-layer linear network $F(x) = UVx$, the important symmetry structure lies in the $GL(h)$ orbits $(U,V) \\to (Ug, g^{-1}V)$, not in fixed points\n\n2. **Overly restrictive theoretical framework**:\n   - The condition $P\\theta' = \\theta'$ for $(\\theta', P)$-reflection symmetry is artificially restrictive\n   - Simple examples show valid reflections that violate this condition\n   - For instance, in 2D with $\\theta' = (0,1)$ and reflection in x-direction $P = ((1,0),(0,0))$, we have $P\\theta' \\neq \\theta'$ yet this is a perfectly valid reflection symmetry\n\n3. **Questionable arguments about gradient vanishing**:\n   - The paper suggests symmetric points trap models due to vanishing gradients\n   - However, in the UV example, fixed points along $U=V$ line have nonzero gradients except at origin and minima\n   - Gradients naturally vanish along symmetry orbits (they're level sets of the loss), but this doesn't imply trapping as claimed\n\n4. **Limited analysis of regularization effects**:\n   - The D-norm modification merely changes spherical weight decay to ellipsoidal weight decay\n   - The $\\theta_0$ shift changes the center of the regularization\n   - While these modifications may improve optimization, it's not clear they \"remove symmetries\" as claimed\n   - In the UV example, these modifications may select preferred points along symmetry orbits (like weight decay does) but may not fundamentally break the symmetry structure\n\n5. **Experimental limitations**:\n   - Lack of systematic comparison with standard weight decay baseline\n   - Only a few experiments (particularly continual learning) show clear improvements over weight decay\n   - Results suggest their method might be a useful variant of weight decay, but not necessarily for the theoretical reasons provided\n\nThe paper's suggested regularization may indeed have advantages over standard weight decay in certain scenarios, but the theoretical arguments about symmetry breaking are not convincing, and the empirical validation needs stronger baselines to support the claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yX965WS95g", "reviewer": "Reviewer_akeV", "summary": "This paper addresses the problem of low-capacity states, often referred to as \"collapse,\" that arise when neural network loss functions contain inherent symmetries. The authors prove two mechanisms through which symmetries reduce model expressivity, leading to these collapsed states and propose a method called syre to eliminate such symmetries. The syre algorithm introduces static random biases alongside weight decay, providing a simple yet effective means to remove almost all symmetry-induced constraints without requiring knowledge of the specific symmetries. The method is claimed to be model-agnostic, applicable to a broad range of architectures, and effective in both theoretical and practical contexts. Empirical results support the effectiveness of the proposed method in diverse settings, including supervised learning, self-supervised learning, and continual learning.", "questions": "- How does syre scale to large models like transformers? Have you tested its efficiency or computational impact on such architectures?\n- Would an adaptive bias approach improve the method compared to using a static bias throughout training?\n- How sensitive is syre to hyperparameter choices, and do you have heuristics to guide practitioners in tuning them?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents a novel approach to tackling symmetry-induced collapses in neural networks. The proposed method uses static random biases to remove symmetries, a simple yet innovative solution. The theoretical insights connect symmetry and expressivity in a new way, offering a significant advancement.\n- The theoretical contributions are rigorous and well-supported with proofs. The authors also provide a range of empirical validations that align with their theoretical claims, making a strong case for the method’s effectiveness across various scenarios.\n- The paper is clearly structured, with a logical flow from problem setup to solution. The core concepts are well-explained.\n- The method's simplicity and general applicability make it impactful. By addressing a fundamental problem in deep learning training, syre has potential benefits for diverse architectures and tasks, opening new research avenues into model expressivity and optimization.", "weaknesses": "- Experiments are limited to small- or medium-scale models. The method’s applicability to real-world, large-scale tasks is insufficiently demonstrated. Benchmarks on high-impact datasets, such as ImageNet, and large-scale architectures, like transformers, would add credibility.\n- The impact of completely removing symmetries on generalization is not well-discussed. Symmetry sometimes acts as a regularizer, and the absence of analysis on this potential downside is a gap.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)