Remove Symmetries to Control Model Expressivity

## Abstract

Abstract When symmetry is present in the loss function, the model is likely to be trapped in a low-capacity state that is sometimes known as a “collapse.” Being trapped in these low-capacity states can be a major obstacle to training across many scenarios where deep learning technology is applied. We first prove two concrete mechanisms through which symmetries lead to reduced capacities and ignored features during training. We then propose a simple and theoretically justified algorithm, syre , to remove almost all symmetry-induced low-capacity states in neural networks. The proposed method is shown to improve the training of neural networks in scenarios when this type of entrapment is especially a concern. A remarkable merit of the proposed method is that it is model-agnostic and does not require any knowledge of the symmetry.

## 1 Introduction

The unprecedented scale and complexity of modern neural networks, which incorporate a vast number of neurons and connections, inherently introduce a high degree of redundancy in model parameters. This complexity in the architecture and the design of loss functions often implies that the loss functions are invariant to various hidden, nonlinear, and nonlocal transformations of the model parameters. These invariant transformations, or “symmetries,” in the loss function have been extensively documented in the literature, with common examples including permutation, rescaling, scale, and rotation symmetries [, , , , , , , , , ].

A unifying perspective to understand how these symmetries affect learning is the framework of reflection symmetries [].(^1^11Essentially, this is because the common symmetries in deep learning all contain $\mathbb{Z}_{2}$ as a subgroup.) A per-sample loss function $\ell$ possesses a $P$-reflection symmetry if for all $\theta$, and all data points $x$,

$$ $\ell((I-2P)\theta,x)=\ell(\theta,x).$ (1) $$

The solutions where $\theta=(I-P)\theta$ are the symmetric solutions with respect to $P$. It has been shown that almost all contemporary models contain quite a few reflection symmetries. Permutation, rescaling, scale, and rotation symmetries all imply the existence of one or multiple reflection symmetries in the loss.

Recent literature has shown that symmetries in the loss function of neural networks often lead to the formation of low-capacity saddle points within the loss landscape [, , ]. These saddle points are located at the symmetric solutions and often possess a lower capacity than the minimizers of the loss []. When a model encounters these saddle points during training, the model parameters are not only slow to escape them but also attracted to these solutions because these the gradient noise also vanish close to these saddles [, ]. Essentially, the model’s learning process stagnates, and it fails to achieve optimal performance due to reduced capacity. However, while many works have characterized the dynamical properties of training algorithms close to symmetric solutions, no methods are known to enable full escape from them.

Because these low-capacity saddles are created by symmetries, we propose a method to explicitly remove these symmetries from the loss functions of neural networks. The method we propose is theoretically justified and only takes one line of code to implement. By removing these symmetries, our method allows neural networks to explore a more diverse set of parameter spaces and access more expressive solutions. The main contributions of this work are:

- 1.
We show how discrete symmetries in the model can severely limit the expressivity of neural networks in the form commonly known as “collapses” (Section 4);
- 2.
We propose a simple method that provably removes almost all symmetries in neural networks, without having any knowledge of the symmetry (Section 5);
- 3.
We apply the method to solve a broad range of practical problems where symmetry-impaired training can be a major concern (Section 6).

We introduce the notations and problem setting for our work in the next section. Closely related works are discussed in Section 3. All the proofs and experimental details are deferred to the appendix.

## 2 Discrete Symmetries in Neural Networks

#### Notation.

For a matrix $A$, we use $A^{+}$ to denote the pseudo-inverse of $A$. For groups $U$ and $G$, $U\lhd G$ denotes that $U$ is a subgroup of $G$. For a vector $w$ and matrix $D$, $\|\theta\|^{2}_{D}:=w^{T}Dw$ is the norm of $w$ with respect to $D$. $\odot$ denotes the element-wise product between vectors.

Let $f(\theta,x)$ be a function of the model parameters $\theta$ and input data point $x$. For example, $f$ could either be a sample-wise loss function $\ell$ or the model itself. Whenever $f$ satisfies the following condition, we say that $f$ has the $P$-reflection symmetry (general symmetry groups are dealt with in Theorem 5 in Section 5).

###### Definition 1 .

Let $P$ be a projection matrix and $\theta^{\prime}$ be a point. $f$ is said to have the $(\theta^{\prime},P)$-reflection symmetry if for all $x$ and $\theta$, (1) $f(\theta+\theta^{\prime},x)=f((I-2P)\theta+\theta^{\prime},x)$, and (2) $P\theta^{\prime}=\theta^{\prime}$.

The second condition is due to the fact that there is a redundancy in the choice of $\theta^{\prime}$ when $\theta^{\prime}\neq 0$. Requiring $P\theta^{\prime}=\theta^{\prime}$ removes this redundancy and makes the choice of $\theta^{\prime}$ unique. Since every projection matrix can be written as a product of a (full-rank or low-rank) matrix $O$ with orthonormal columns, one can write $P=OO^{T}$ and refer to this symmetry as an $O$ symmetry. In common deep learning scenarios, it is almost always the case that $\theta^{\prime}=0$ (for example, this holds for the common cases of rescaling symmetries, (double) rotation symmetries, and permutation symmetries, see Theorem 2-4 of Ref. []). A consequence of $\theta^{\prime}=0$ is that the symmetric projection $P\theta$ of any $\theta$ always has a smaller norm than $\theta$: thus, a symmetric solution is coupled to the solutions of weight decay, which also favors small-norm solutions. As an example of reflection symmetry, consider a simple tanh network $f(\theta,x)=\theta_{1}\tanh(\theta_{2}x)$. The model output is invariant to a simultaneous sign flip of $\theta_{1}$ and $\theta_{2}$. This corresponds to a reflection symmetry whose projection matrix is the identity $P=((1,0),(0,1))$. The symmetric solutions correspond to the trivial state where $\theta_{1}=\theta_{2}=0$.

To be more general, we allow $\theta^{\prime}$ to be nonzero to generalize the theory and method to generic hyperplanes since the purpose of the method is to remove all reflection symmetries that may be hidden or difficult to enumerate. Let us first establish some basic properties of a loss function with $P$-reflection, to gain some intuition (again, proofs are in the appendix).

###### Proposition 1 .

Let $f$ have the $(\theta^{\prime},P)$-symmetry and Let $f^{\prime}(\theta)=f(\theta+\theta^{\dagger})$. Then, (1) for any $\theta^{\dagger}$ such that $P\theta^{\dagger}=\theta^{\prime}$, $f(\theta+\theta^{\dagger})=f((I-2P)\theta+\theta^{\dagger})$, and (2) $f^{\prime}$ has the $(\theta^{\prime}-\theta^{\dagger},P)$ symmetry.

Therefore, requiring $P\theta^{\prime}=\theta^{\prime}$ reduces different manifestations of the symmetry to a unique one and simply shifting the function will not remove any symmetry. This proposition emphasizes the subtle difficulty in removing a symmetry.

###### Definition 1 .

###### Proposition 1 .

## 3 Related Works

One closely related work is Ref. [], which shows that every reflection symmetry in the model leads to a low-capacity solution that is energetically favored when weight decay is used. This is because the minimizer of the weight decay is coupled with stationary points of the reflection symmetries – the projection of any parameter to a symmetric subspace always decreases the norm of the parameter, and is thus energetically preferred by weight decay. Our work develops a method to decouple symmetries and weight decay, thus avoiding collapsing into low-capacity states during training. Besides weight decay, an alternative mechanism for this type of capacity loss is gradient-noise induced collapse, which happens when the learning rate - batchsize ratio is high [, ].

Contemporarily, Ref. [] empirically explores how removing symmetries can benefit neural network training and suggests a heuristic for removing symmetries by hold a fraction of the weights unchanged during training. However, the proposed method can only remove permutation symmetries in fully connected layers. This is particularly a problem because most of the symmetries in nonlinear systems are unknown and hidden [, ]. In sharp contrast, the technique proposed in our work is both architecture-independent and symmetry-agnostic and provably removes all known and unknown $P$-reflection symmetries in the loss.

## 4 Symmetry Impairs Model Capacity

We first show that reflection symmetry directly affects the model capacity. For simplicity, we let $\theta^{\prime}=0$ for all symmetries. Let $f(x,\theta)\in\mathbb{R}$ be a Taylor-expandable model that contains a $P$-reflection symmetry. Let $\Delta=\theta-\theta_{0}$. Then, close to any symmetric point $\theta_{0}$ (any $\theta_{0}$ for which $P\theta_{0}=0$), for all $x$, Ref. [] showed that

$$ $\displaystyle f(x,\theta)-f(x,\theta_{0})=$ $\displaystyle\underbrace{\nabla_{\theta}f(x,\theta_{0})(I-P)\Delta+O(\|P\Delta \|)^{2}}_{\text{Symmetry subspace}}+\underbrace{\frac{1}{2}\Delta^{T}PH(x)P \Delta+O(\|\Delta\|^{3})}_{\text{Symmetry-broken subspace}},$ (2) $$

where $H(x)$ is the Hessian matrix of $f$. An important feature is that the symmetry subspace is a generic expansion where both odd and even terms are present, and the first order term does not vanish in general. In contrast, in the symmetry-broken subspace, all odd-order terms in the expansion vanish, and the leading order term is the second order. This implies that close to a symmetric solution, escaping from it will be slow, and if at the symmetric solution, it is impossible for gradient descent to leave it. The effect of this entrapment can be quantified by the following two propositions.

###### Proposition 2 .

(Symmetry removes feature.) Let $f$ have the $P$-symmetry, and $\theta$ be intialized at $\theta_{0}$ such that $P\theta_{0}=0$. Then, the kernalized model, $g(x,\theta)=\lim_{\lambda\to 0}(\lambda^{-1}f(x,\lambda\theta+\theta_{0})-f(x,
\theta_{0}))$, converges to

$$ $\theta^{*}=A^{+}\sum_{x}\nabla_{\theta}f(x,\theta_{0})^{T}y(x)$ (3) $$

under GD for a sufficiently small learning rate. Here $A:=(I-P)\sum_{x}\nabla_{\theta}f(x,\theta_{0})^{T}\nabla_{\theta}f(x,\theta_{0
})(I-P)$ and $A^{+}$ denotes the Moore–Penrose inverse of $A$.

Figure: Figure 1: Training loss $\ell$ as a function of the iteration $t$ for a fully connected network on MNIST. Starting from a low-capacity state, vanilla neural networks are trapped under SGD or Adam training (left). Blacker lines correspond to higher-capacity initializations, where more neurons are away from the permutation-symmetric state. When the symmetries are removed, the capacity of the initialization no longer affects the solution found at the end of the training (right).
Refer to caption: extracted/5817815/figures/training_loss.png

This means that in the kernel regime(^2^22Technically, this is the lazy training limit [].), being at a symmetric solution implies that the feature kernel features are being masked by the projection matrix:

$$ $\nabla_{\theta}f(x,\theta_{0})\to(I-P)\nabla_{\theta}f(x,\theta_{0}),$ (4) $$

and learning can only happen given these masks. This implies that the model is not using the full feature space that is available to it.

###### Proposition 3 .

(Symmetry reduces parameter dimension.) Let $f$ have the $P$-symmetry, and $\theta\in\mathbb{R}^{d}$ be intialized at $\theta_{0}$ such that $P\theta_{0}=0$. Then, for all time steps $t$ under GD or SGD, there exists a model $f^{\prime}(x,\theta^{\prime})$ and sequence of parameters $\theta^{\prime}_{t}$ such that for all $x$,

$$ $f^{\prime}(x,\theta_{t}^{\prime})=f(x,\theta_{t}),$ (5) $$

where ${\rm dim}(\theta^{\prime})=d-{\rm rank}(P)$.

The existence of this type of dimension reduction when symmetry is present has also been noticed by previous works in case of permutation symmetry []. Intuitively, the above two results follow from the fact that the symmetric subspaces of reflection symmetries (and any general discrete symmetries) are a linear subspace, and so gradient descent (and thus gradient flow) cannot take the model away from it, despite the discretization error. This means that, essentially, the model is identical to a model with a strictly smaller dimension throughout its training, no matter whether the training is through GD or SGD. When there are multiple symmetries, there is a compounding effect. An alternative perspective to this problem is through the classical framework of singular learning theories, where each symmetry corresponds to a singularity and directly affects the asymptotic performance of the model [].

See Figure 1 for an illustration. We initialize a two-layer ReLU neural network on a low-capacity state where a fraction of the hidden neurons are identical (corresponding to the symmetric states of the permutation symmetry []) and train with and without removing the symmetries. We see that when the symmetries are removed (with the method proposed in the next section), the model is no longer stuck at these neuron-collapsed solutions.

###### Proposition 2 .

###### Proposition 3 .

## 5 Removing Symmetry with Static Bias

Next, we prove that a simple algorithm that involves almost no modification to any deep learning training pipeline can remove almost all such symmetries from the loss without creating new ones. From this section onward, we will consider the case where the function under consideration is the loss function (a per-batch loss or its expectation): $f=\ell$.

### 5.1 Countable Symmetries

We seek an algorithm that eliminates the reflection symmetries from the loss function $\ell$. We show that when the number of reflection symmetries in the loss function is finite, one can completely remove them using a simple technique. The symmetries are required to have the following property and the loss function is assumed to obey assumption 1.

###### Property 1 .

(Enumeratability)
There exists a countable set of pairs of projection matrices and biases $S=\{(\theta^{\dagger}_{i},P_{i})\}_{i}^{N}$ such that $\ell(\theta,x)$ has the $\theta_{i}^{\dagger}$-centric $P_{i}$-reflection symmetry for all $i$. In addition, $\ell$ does not have any $(\theta^{\dagger},P)$ symmetry for $(\theta,P)\notin S$.

###### Assumption 1 .

There only exists countably many pairs $(c_{0},\tilde{\theta})$ such that $g(x)=\ell(\theta,x)-c_{0}\theta$ contains a $\tilde{\theta}$-centric $P$ symmetry, where we require $Pc_{0}=c_{0}$ and $P\tilde{\theta}=\tilde{\theta}$.

This assumption is satisfied by common neural networks with standard activations. The main purpose of this assumption is to rule out the pathological of a linear or quadratic deterministic objective, which never appears in practice or for which symmetry is not a concern.(^3^33An example that violates this assumption is when $\ell(\theta,x)=c_{0}^{T}\theta$. $\ell(\theta,x)-c_{0}^{T}\theta$ has infinitely many reflection symmetries everywhere and for every data point because it is a constant function.)

For symmetry removal, we propose to utilize the following alternative loss function. Let $\theta_{0}$ be drawn from a Gaussian distribution with variance $\sigma_{0}$ and $\ell$ be the original loss function:

$$ $\ell_{\rm r}(\theta,x)=\ell(\theta+\theta_{0})+\gamma\|\theta\|^{2}.$ (6) $$

$\gamma$ is nothing but the standard weight decay. We will see that using a static bias along with weight decay is essential for the method to work. We find that with unit probability, the loss function $\ell_{\rm r}$ contains no reflection symmetry:

###### Theorem 1 .

Let $\ell$ satisfy Property 1 and Assumption 1. Then, with probability $1$ (over the sampling of $\theta_{0}$), there exists no projection matrix $P$ and reflection point $\theta^{\prime}$ such that $\ell_{\rm r}$ has the $(\theta^{\prime},P)$-symmetry.

The core mechanism of this theorem is decoupling of the solutions of the symmetries from the solutions of the weight decay. With weight decay, a solution with a small norm is favored, whereas with a random bias, the symmetric solutions are shifted by a small constant and no longer overlap with solutions that have a small norm. Equivalently, this method can be implemented as a random bias in the $L_{2}$ regularization: $\ell_{\rm r}(\theta,x)=\ell(\theta)+\gamma\|\theta+\theta_{0}\|^{2}$. This is a result of the fact that simple translation does not change the nature of the function. There are two remarkable parts of the theorem: (1) it not only removes all existing symmetries but also guarantees that there are no remaining ones; (2) the proof works as long as $\theta_{0}$ is nonzero. This hints at the possibility of using a small and random $\theta_{0}$, which removes all symmetries in principle and also does not essentially affect the solutions of the original objective. In this work, we will refer to the method in Eq. (6) as syre, an abbreviation for “symmetry removal.”

###### Property 1 .

###### Assumption 1 .

###### Theorem 1 .

### 5.2 Uncountably Many Symmetries

In deep learning, it is possible for the model to simultaneously contain infinitely many reflection symmetries. This happens, for example, when the model parameters have the rotation symmetry or the double rotation symmetry (common in self-supervised learning problems or transformers). It turns out that simply adding a static bias and weight decay is not sufficient to remove all symmetries in this case.

We propose to train on the following alternative loss instead, where $ar$ stands for “advanced removal”:

$$ $\ell_{\rm ar}(\theta)=\ell(\theta+\theta_{0})+\gamma\|\theta\|_{D}^{2},$ (7) $$

where $D$ is a positive diagonal matrix in which all diagonal elements of $D$ are different. The simplest way to achieve such a $D$ is to set $D_{ii}\sim{\rm Uniform}(1-\epsilon,1+\epsilon)$, where $\epsilon$ is a small quantity.

###### Theorem 2 .

Any $(\theta^{\prime},P)$-symmetry that $\ell_{\rm ar}$ satisfies obeys: (1) $P\theta_{0}=\theta^{\prime}$ (2) and $PD=DP$.

Conditions (1) and (2) implies that there are at most finitely many $(\theta^{\prime},P)$-symmetry $\ell_{\rm ar}$ can have. When there does not exist any symmetry that satisfies this condition, we have removed all the symmetries. In the worst case where $\ell$ is a constant function, there are $2^{N}$ symmetries where $N$ is the number of reflection symmetries. If we further assume that every $P$ is associated with at most finitely many $\theta^{\prime}$, then we, again, remove all symmetries with probability $1$. The easiest way to determine this $D$ matrix is through sampling from a uniform distribution with a variance $\sigma_{D}\ll 1$.

###### Theorem 2 .

### 5.3 Strength of Symmetry Removal

While any level of $\theta_{0}$ and $\sigma_{D}$ are sufficient to remove the symmetries, one might want to quantify the degree to which the symmetries are broken. This is especially relevant when the model is located close to a symmetric solution and requires a large gradient to escape from it. Also, a related question that may arise in practice is how large one should choose $\sigma_{0}$ and $\sigma_{D}$, which are the variances of $\theta_{0}$ and $D$. The following theorem gives a quantitative characterization of the degree of symmetry breaking.(^4^44Because $\theta_{0}$ and $D$ are randomly sampled, the big-$O$ notation $x\in O(z)$ is used to mean that as the scaling parameter tends to infinity, $\exists c_{0}$ such that $\Pr(\|x\|<c_{0}z)\to 1$.)

###### Theorem 3 .

Let the original loss satisfy a $(\theta^{*},P)$-symmetry, where $\theta^{*}\in\mathbb{R}^{d}$. Then, for any local minimum $\theta\in\Theta(1)$, if $\sigma_{D}=o(\sigma_{0})$ and $P\theta\neq 0$,

$$ $\Delta:=\frac{1}{\|P\theta\|}\left[{\ell_{\rm ar}(\theta+\theta^{*})-\ell_{\rm ar }((I-2P)\theta+\theta^{*})}\right]=\Omega(\gamma\sigma_{0}).$ (8) $$

This theorem essentially shows a “super-Lipschitz” property of the difference between the loss function values between parameters and their reflections. This means that with a random bias, the symmetry will be quite strongly removed, as long as we ensure $\sigma_{D}\ll\sigma_{0}$, which is certainly ensured when $\sigma_{D}=0$. As a corollary, it also shows that after applying a static bias, no symmetric solution where $P\theta=0$ can still be a stationary point because as $P\theta\to 0$, the quantity $\Delta$ converges to the projection of the gradient onto symmetry breaking subspace.

###### Corollary 1 .

For any $\theta$ such that $P\theta=0$, $P\nabla_{\theta}\ell_{\rm ar}=\Omega(\gamma\sigma_{0})$.

Now, the more advanced question is the case when there are multiple reflection symmetries, and one wants to significantly remove every one of them.

###### Theorem 4 .

Let $\ell$ contain $N$ reflection symmetries: $\{(P_{i},\theta^{*}_{i})\}_{i=1}^{N}$. Let

$$ $\Delta_{i}:=\frac{{\ell_{\rm ar}(\theta+\theta_{i}^{*})-\ell_{\rm ar}((I-2P_{i })\theta+\theta_{i}^{*})}}{\|P_{i}\theta\|}.$ (9) $$

Then, for any local minimum $\theta\in\Theta(1)$, letting $\gamma\sigma_{0}=\Omega\left(\frac{2\epsilon N}{1-\delta}\right)$ guarantees that $\Pr(\min_{i}|\Delta_{i}|>\epsilon)>\delta$ for any $\epsilon$ and $\delta<1$.

In the theorem, the probability is taken over the random sampling of the static bias.
Namely, the achievable strengths of symmetry-breaking scales inversely linearly in $N$, the size of the minimal set of the entire group generated by $N$ reflections.
In general, without further assumptions there is no way to improve this scaling because, for example, the smallest of $N$ independent bounded variables roughly scales as $1/N$ towards its lower boundary.

###### Theorem 3 .

###### Corollary 1 .

###### Theorem 4 .

#### General Groups.

Lastly, one can generalize the theory to prove that the proposed method removes symmetries from a generic group. Let $G$ be the linear representation of a generic finite group, possibly with many nontrivial subgroups. If the loss function $\ell$ is invariant under transformation by the group $G$, then

$$ $\forall g,\ \ell(\theta)=\ell(g\theta).$ (10) $$

Because $G$ is finite, it follows that the representations $g$ must be full-rank and unipotent.

The following theorem shows that at every symmetric solution, there exists an escape direction with a strong gradient that pulls the parameters from every subgroup of the related symmetry. In other words, it is no longer possible to be trapped in a symmetric solution. While general groups appear less common in deep learning scenarios, they exist in a lot of application scenarios with equivariant networks [], where it is common to incorporate generic group structures into the model.

Letting $U$ be a subgroup of $G$, we denote with the overbar the following matrix:

$$ $\overline{U}=\frac{1}{|U|}\sum_{u\in U}u.$ (11) $$

Note that $\overline{U}$ is a projection matrix: $\overline{U}\,\overline{U}=\overline{U}$. This means that $I-\overline{U}$ is also a projection matrix. Importantly, $\overline{U}$ projects a vector into the symmetric subspace. For any $u\in U$, $u\overline{U}=\overline{U}$.
Likewise, $I-\overline{U}$ projects any vector into the symmetry-broken subspace, a well-known result in the theory of finite groups []. We denote by

$$ $\Delta_{V}=\|(I-\overline{V})\nabla_{\theta}\ell\|$ (12) $$

the strength of the symmetry removal for the subgroup $V$.

###### Theorem 5 .

Let $\Gamma(G)$ denote the smallest minimal generating set for the group $G$. $Z$ denotes the number of minimal subgroups of $G$. Let $\ell$ be invariant under the group transformation $G$ and let $\theta$ be in the invariant subspace of a subgroup $U\lhd G$. Then, for every subgroup $V\lhd U\lhd G$,

- 1.
$\Delta_{V}=\Omega(\gamma\sigma_{0}{\rm rank}(I-\overline{V}))$;
- 2.
$\min_{V\lhd U}\Delta_{V}=\Omega(\gamma\sigma_{0}{\rm rank}(I-\overline{V})Z^{-
1})$;
- 3.
if $G$ is abelian, $\min_{V\lhd U}\Delta_{V}=\Omega(\gamma\sigma_{0}{\rm rank}(I-\overline{V})|
\Gamma(U)|^{-1})$;(^5^55This result can be generalized to the case where all projectors $\bar{V}$ commute with each other, even if $G$ is nonabelian. Essentially, this result is a consequence of the properties of the representations of $G$ and not due to $G$ itself.)
- 4.
additionally, for any $\epsilon>0$ and $\delta<1$, $\Pr(\min_{V\lhd U}\Delta_{V}>\epsilon)>\delta$, if $\gamma\sigma_{0}=\Omega\left(\frac{2\epsilon|\Gamma(U)|}{1-\delta}\right)$.

Item (2) of the theorem is essentially due to the fact that removing symmetries from a larger group can be reduced to removing them from one of its subgroups. Conceptually, this result has the same root as the classical theory of combinatorial designs, where averaging over a subset of groups has the same effect as averaging over the whole group []. In general, $1\leq Z\leq|G|$ (and sometimes $\ll|G|$), and so this scaling is not bad. For the part (3) of the theorem, the term $|\Gamma(U)|$ is especially meaningful. It is well-known that $|\Gamma(U)|\leq\log|U|$, and so the worst-case symmetry-breaking strength is only of order $1/\log|U|$, which is far slower than what one would expect. In fact, for a finite group with size $N$, the number of subgroups can grow as fast as $N^{\log N}$ [], and thus, one might naively think that the minimal breaking strength decreases as $N^{-\log N}$. This theorem shows that the proposed method is highly effective at breaking the symmetries in the loss function or the model.

Figure: Figure 2: With the proposed method, the robustness of the feature increases with the input dimension. The same may happen under vanilla training, but the effect is not strong. The covariance matrix is measured from the first-layer output with batch size $1000$, and we set eigenvalues larger than $10^{-4}$ to zero.
Refer to caption: extracted/5817815/figures/input_dim.png

A numerical example is shown in Figure 2, which validates a major prediction of the theorem: a symmetry is easier to remove if it is high-dimensional. We train a two-layer ReLU net in a teacher-student scenario and change the input dimension. This experiment holds the number of (permutation) symmetries fixed and directly controls ${\rm rank}(I-\overline{V})$. As the input dimension increases, the symmetry of the learned model becomes lower. In comparison, without a static bias, having a high dimension is not so helpful.

###### Theorem 5 .

#### Role of Noise.

A core concept in our theory is that random noise, even if static, can help remove the symmetries. A related question is whether simply having noise in the input or label is sufficient, and so no static bias is required. The answer is no. In Appendix B.1, we find that adding input noise does help alleviate collapse, whereas adding label noise exacerbates it.

### 5.4 Hyperparameter and Implementation Remark

As discussed, there are two ways to implement the method (Eq. (6) or Eq. (7)). In our experiments, we stick to the definition of Eq. (6), where the model parameters are biased, and weight decay is the same as the standard implementation. For the choice of hyperparameters, we always set $\sigma_{D}=0$ as we find only introducing $\sigma_{0}$ to be sufficient for most tasks. Experiments with standard training settings (see the next section for the Resnet18 experiment on CIFAR-10) show that choosing $\sigma_{0}$ to be at least an order of magnitude smaller than the standard initialization scale (usually of order $1/\sqrt{d}$ for a width of $d$) works the best. We thus recommend a default value of $\sigma_{0}$ to be $0.01/\sqrt{d}$, where $1/d$ is the common initialization variance. For the rest of the paper, we state $\sigma_{0}$ in relative units of $\sqrt{d}^{-1}$ for this reason. That being said, we stress that $\sigma_{0}$ is a hyperparameter worth tuning, as it directly controls the tradeoff between optimization and symmetry removal.

## 6 Experiment

First, we show that the proposed method is compatible with standard training methods. We then apply the method to a few settings where symmetry is known to be a major problem in training. We see that the algorithm leads to improved model performance on these problems.

### 6.1 Compatibility with Standard Training

#### Ridge linear regression.

Let us first consider the classical problem of linear regression with $d$-dimensional data, where one wants to find $\min_{w}\sum_{i}(w^{T}x_{i}-y_{i})^{2}$. Here, the use of weight decay has a well-known effect of preventing the divergence of generalization loss at a critical dataset size $N=d$ [, ]. This is due to the fact that the Hessian matrix of the loss becomes singular exactly at $N=d$ (at infinite $N$ and $d$). The use of weight decay shifts all the eigenvalues of the Hessian by $\gamma$ and removes this singularity. In this case, one can show that the proposed method is essentially identical to the simple ridge regression. The ridge solution is $w^{*}=\mathbb{E}[\gamma I+A]^{-1}\mathbb{E}[xy]$, where $A=\mathbb{E}[xx^{T}]$, and the solution to the biased model is

$$ $w^{*}=\mathbb{E}[\gamma I+A]^{-1}(\mathbb{E}[xy]+\gamma\theta_{0}).$ (13) $$

The difference is negligible with the original solution if either $\gamma$ and $\theta_{0}$ are small. See Figure 3-left.

Figure: Figure 3: Training with syre in standard settings. The result shows that biasing the models by a small static bias does not change the performance of standard training settings. Left: Application of the method to a linear regression problem. Here, $\alpha^{-1}=d/N$ is the degree of parameterization. A well-known use of weight decay is to prevent double descent when $\alpha=1$. Here, we see that the proposed method works as well as vanilla weight decay. Because there is no reflection symmetry in the problem, the proposed method should approximate vanilla weight decay. Mid: Test accuracy of Resnet18 on the CIFAR-10 datasets. The blue line denotes the performance of Resnet, and the shadowed area denotes its standard deviation estimated over $10$ trials. For $\sigma_{0}<0.2$, there is no significant difference between the performance of the vanilla Resnet and syre Resnet. Right: linear regression with a redundant parametrization [, ]. The loss function takes the form $\ell(u,w)=\|(u\odot w)^{T}x-y\|^{2}$. Due to symmetry, the point $(u_{i},w_{i})=0$ is a low-capacity state where the $i$-th neuron is “dead”. Training with style, the model stayed away from any trapping low-capacity state during training. In comparison, training with vanilla SGD or a heuristic for fixing the weights does not fix the problem of collapsing to a low-capacity state.
Refer to caption: extracted/5817815/figures/linear_regression.png

#### Reparametrized Linear Regression.

A minimal model with emergent interest in the problem of compressing neural networks is the reparametrized version of linear regression [], the loss function of which is $\ell(u,w)=\|(u\odot w)^{T}x-y\|^{2}$, where we let $u,\ w,\ x\in\mathbb{R}^{200}$ and $y\in\mathbb{R}$. Due to the rescaling symmetry between every parameter $u_{i}$ and $w_{i}$, the solutions where $u_{i}=w_{i}=0$ is a low-capacity state where the $i$-th neuron is “dead.” For this problem, we compare the training with standard SGD and syre. We also compare with a heuristic method (W-fix), where a fraction $\phi=0.3$ of weights of every layer is held fixed with a fixed variance $\kappa=0.01$ at initialization. This method has been suggested in [] as a heuristic for removing symmetries and is found to work well when there is permutation symmetry. We see that both the vanilla training and the W-fix collapse to low-capacity states during training, whereas the proposed method stayed away from them throughout. The reason is that the proposed method is model-independent and symmetry-agnostic, working effectively for any type of possibly unknown symmetry in an arbitrary architecture.

#### ResNet.

We also benchmark the performance of the proposed method for ResNet18 with different $\sigma_{0}$ on the CIFAR-10 datasets in Figure 3. When $\sigma_{0}=0$, the syre model is equivalent to the vanilla model. Figure 3 shows that the difference in performance between the vanilla Resnet and syre Resnet is very small and becomes neglectable when $\sigma_{0}<0.2$. We thus recommend a default value of $\sigma_{0}\leq 0.01/\sqrt{d}$.

### 6.2 Benchmarking Symmetry Removal

Figure: Figure 4: The degree of symmetry versus the objective value for two choices of $B$ and various training methods with different hyperparameters. The proposed method is the only method to smoothly interpolate between optimized solutions and solutions with low symmetry. syre performs well in both cases. Left: objective with unstructured symmetry Right: structured symmetry.
Refer to caption: extracted/5817815/figures/benchmark1.png

In this section, we benchmark the effect of symmetry control of the proposed method for two controlled experiments.
To compare the influence of syre and other training methods on the degree of symmetry, we consider minimizing the following objective function

$$ $(w^{T}w)^{2}-w^{T}Bw:=(w^{T}w)^{2}-\sum_{i=1}^{d}\lambda_{i}(v_{i}^{T}w)^{2},$ $$

Benchmarking
where $w\in\mathbb{R}^{d}$ is the optimization parameter and $B\in\mathbb{R}^{d\times d}$ is a given symmetric matrix with eigenvalues $\lambda_{i}$ and eigenvectors $v_{i}$ ($v_{i}^{T}v_{i}=1$). The objective function has $n$ reflection symmetries $P_{i}w:=w-2(v_{i}^{T}w)v_{i}$. Hence, we define the degree of symmetry as $\sum_{i=1}^{d}\mathbf{1}\{v_{i}^{T}w<c_{\rm th}\}$, where $c_{\rm th}$ is a given threshold. Depending on the spectrum of $B$, the nature of the task is different. We thus consider two types of spectra: (1) an unstructured spectrum where $B=G+G^{T}$ for a Gaussian matrix $G$, and (2) a structured spectrum where $B={\rm diag}(v)$ where $v$ is a random Gaussian vector. Conceptually, the first type is more similar to rotation and double rotation symmetries in neural networks where the basis can be arbitrary, while the second is a good model for common discrete symmetries where the basis is often diagonal or sparse. For the first case we choose $c_{\rm th}=10^{-3}$ and for the second case we choose $c_{\rm th}=10^{-1}$.

In Figure 4, we compare syre, W-fix, drop out, weight decay, and the standard training methods in this setting for $d=1000$ and two choices of $B$. In both cases, we use Gaussian initialization and gradient descent with a learning rate of $10^{-4}$. For syre and weight decay, we choose weight decay from $0.1$ to $10$. For W-fix, we choose $\phi$ from $0.001$ to $0.1$. For dropout, we choose a dropout rate from $0.01$ to $0.6$. Figure 4 shows that for both cases, syre is the only method that effectively and smoothly interpolates between solutions with low symmetry and best optimization. This is a strong piece of evidence that the proposed method can control the degree of symmetries in the model.

### 6.3 Feature and Neuron Collapses in Supervised Learning

Figure: Figure 5: Performance of a 4-layer FCN for datasets with various weight decay $\gamma$ and data distributions with varyings strengths of linear correlation $\alpha$. As the theory predicts, the covariance of the vanilla model has a low-rank structure and performs significantly worse. In the main figures, solid lines denote training accuracy and dashed lines denote test accuracy. The dashed black line corresponds to random guess. Subfigures show the rank of the covariance matrix of the first layer output before (solid lines) and after (dashed lines) activation with batch size $1000$ scaled by $\alpha^{2}$. We set eigenvalues smaller than $10^{-4}$ to 0 0. Left: $\alpha=1$ and different $\gamma$. Right: $\gamma=0.01$ and different $\alpha$.
Refer to caption: extracted/5817815/figures/4layer.png

See Figure 5, where we train the vanilla and syre four-layer networks with various levels of weight decay $\gamma$ and various levels of input-output covariance $\alpha$. The dataset is constructed by rescaling the input by a factor of $\alpha$ for the MNIST dataset. The theory predicts that the syre model can remove the permutation symmetry in the hidden layer. This is supported by subfigures in Figure 5, where vanilla training results in a low-rank solution. Meanwhile, the accuracy of the low-rank solution is significantly lower for a large $\gamma$ or a small $\alpha$, which corresponds to the so-called neural collapses. Also, we observe that syre shifts the eigenvalues of the representation by a magnitude proportional to $\sigma_{0}$, thus explaining the robustness of the method against collapses in the latent representation (See Figure 11).

### 6.4 Posterior Collapse in Bayesian learning

Ref. [] points out that a type of posterior collapse in Bayesian learning [, ] is caused by the low-rankness of the solutions. We show that training with syre could overcome this kind of posterior collapse. In Figure 6, we train a $\beta$-VAE [, ] on the Fashion MNIST dataset. Following Ref. [], we use $\beta$ to weigh the KL loss, which can be regarded as the strength of prior matching. Both the encoder and the decoder are a two-layer network with SiLU activation. The hidden dimension and the latent dimension are $200$. Only the encoder has weight decay because the low-rank problem is caused by the encoder rather than the decoder. We also choose the prior variance of the latent variable to be $\eta_{enc}=0.01$. Other settings are the same as []. Posterior collapse happens at $\beta=10$, signalized by a large reconstruction loss in the right side of Figure 6. However, the reconstruction loss decreases, and the rank of the encoder output increases (according to the left side of Figure 6) after we use weight decay and syre. This is further verified by the generated image in Figure 7. Therefore, we successfully remove the permutation symmetry of the encoder.

Figure: Figure 6: Rank and reconstruction loss for a VAE is trained on the Fashion MNIST dataset. The covariance of the vanilla model has a low-rank structure and larger reconstruction loss. More importantly, posterior collapse happens at $\beta=5$ but is mitigated with weight decay. Left: the rank of the encoder output with batch size $1000$. We set eigenvalues smaller than $10^{-6}$ to 0 0. Right: reconstruction loss of vanilla and syre models.
Refer to caption: extracted/5817815/figures/VAE_rank.png

Figure: Figure 7: Examples of Fashion MNIST reconstruction with syre and $\beta=10$. Left: No weight decay. Right: $\gamma=1000$. Clearly, the posterior collapse is mitigated by imposing syre with weight decay.
Refer to caption: extracted/5817815/figures/fashion_VAE_weight_decay0.png

**Table 1: Performance of the linearly evaluated Resnet18 on CIFAR100 for the unsupervised self-constrastive learning task. Here, the low-rankness measures the proportion of eigenvalues smaller than $10^{-5}$. Our experiment indicates that symmetry-induced reduction in model capacity can explain about $50\%$ of the performance difference between the representation of the two layers.**
|  | Hyperparameter | Low-rankness | Penult. Layer Acc. | Last Layer Acc. |
| --- | --- | --- | --- | --- |
| vanilla [] | - | $70\%$ | $46.8\%$ | $22.2\%$ |
| syre | $\sigma_{0}=0.1$ | $0\%$ | $46.8\%$ | $31.7\%$ |
|  | $\sigma_{0}=0.01$ | $0\%$ | $46.2\%$ | $32.5\%$ |
|  | $\sigma_{0}=0.1$, all layers | $0\%$ | $44.6\%$ | $30.7\%$ |
|  | $\sigma_{0}=0.01$, all layers | $0\%$ | $45.4\%$ | $32.4\%$ |

### 6.5 Low-Capacity Trap in Self-supervised Learning

A common but bizarre practice in self-supervised learning (SSL) is to throw away the last layer of the trained model and use the penultimate learning representation, which is found to have much better expressiveness than the last layer representation. From the perspective of symmetry, this problem is caused by the rotation symmetry of the last weight matrix in the SimCLR loss []. We train a Resnet-18 together with a two-layer projection head over the CIFAR-100 dataset according to the setting for training SimCLR in []. Then, a linear classifier is trained using the learned representations. Our implementation reproduces the typical accuracy of SimCLR over the CIFAR-100 dataset []. As in [], the hidden layer before the projection head is found to be a better representation than the layer after. Therefore, we apply our syre method to the projection head or to all layers. According to Table 1, syre removes the low-rankness of the learned features and increases the accuracy trained with the features after projection while not changing the representation before projection. Thus, symmetry-induced reduction in model capacity can explain about $50\%$ of the performance difference between the representation of the two layers. Also, an interesting observation is that just improving the expressivity of the last layer is insufficient to close the gap between the performance of the last layer and the penultimate layer. This helps us gain a new insight: symmetry is not the only reason why the last layer representation is defective.

Figure: Figure 8: Performance and accuracy of a CNN trained on a continual learning task (permuted MNIST [, ]). The main figure shows the test accuracy, and the subfigure shows the rank of the convolution layers output with batch size $1000$, where we set eigenvalues smaller than $10^{-4}$ to 0 0. The results suggest that the rank of the vanilla model gradually decreases, and the model completely collapses after the sixth task, while the syre model remains unaffected.
Refer to caption: extracted/5817815/figures/CNN.png

Figure: Figure 9: Loss of plasticity in continual learning in an RL setting. We use the PPO algorithm [] to solve the Slippery-Ant problem []. The rank and the performance of the vanilla PPO decrease quickly, while the rank and the performance of PPO with syre remain the same, beyond that of PPO with weight decay. Left: the effective rank of the policy network as defined in []. Right: returns. Each trajectory is averaged over $5$ different random seeds.
Refer to caption: extracted/5817815/figures/RL_rank.png

### 6.6 Loss of Plasticity in Continual Learning

A form of low-capacity collapse also happens during continual learning, i.e., the plasticity of the network gradually decreases as the model is trained on more and more tasks. This problem is common in both supervised and reinforcement learning settings and may also be relevant to the finetuning of large language models [, , , ].

In Figure 8, we train a CNN with two convolution layers ($10$ channels and $20$ channels) and two fully connected layers ($320$ units and $50$ units) over the MNIST datasets. For the data, we randomly permute the pixels of the training and test sets for $9$ times, forming $10$ different tasks (including the original MNIST). We then train a vanilla CNN and a syre CNN over the $10$ tasks continually with SGD and weight decay $0.01$. The inset of Figure 8 shows that the rank of the original model gradually decreases, but the syre model remains close to full rank. Correspondingly, in the right side of Figure 8, the accuracy over the test set drops while the rank of the original model collapses, but the accuracy of the syre model remains similar.

#### Reinforcement Learning.

In Figure 9, we use the PPO algorithm for the Slippery-Ant problem [], a continual variant of the Pybullet’s Ant problem [] with friction that changes every $5M$ steps. Hyperparameters for the PPO algorithm are borrowed from [], and we use a weight decay of $0.002$ for both PPO with weight decay and with syre. Figure 9 suggests that syre is effective in maintaining the rank of the model during continual training and obtains better performance than pure weight decay.

## 7 Conclusion

Symmetry-induced neural-network training problems exist extensively in machine learning. We have shown that the existence of symmetries in the model or loss function may severely limit the expressivity of the trained model. We then developed a theory that leverages the power of representation theory to show that adding random static biases to the model, along with weight decay, is sufficient to remove almost all symmetries, explicit or hidden. We have demonstrated the relevance of the method to a broad range of applications in deep learning, and a possible future direction is to deploy the method in large language models, which naturally contain many symmetries.
Lastly, it is worth noting that on its own, symmetry is neither good nor bad. For example, practitioners may be interested in introducing symmetries to the model architecture in order to control the capacity of the model []. However, with too much symmetry, the training of models becomes slow and likely to contain many low-capacity traps. Meanwhile, a model completely without symmetry may have undesirably high capacity and be more prone to overfitting. Having the right degree of symmetry might thus be crucial for achieving both smooth optimization and good generalization. With our proposed method, it becomes increasingly possible to deliberately fine-grain engineer symmetries in the loss function, introducing desired symmetries and removing undesirable ones.

## Acknowledgements

The authors acknowledge support from NTT Research, and from the Institute for Artificial Intelligence and Fundamental Interactions (IAIFI) through NSF Grant No. PHY-2019786.

## Appendix A Theoretical Concerns

### A.1 Proof of Proposition 1

###### Proof.

Part (1). Note that we have

$$ $\theta^{\dagger}=P\theta^{\dagger}+(I-P)\theta^{\dagger}=\theta^{\prime}+(I-P) \theta^{\dagger}.$ (14) $$

Thus,

$$ $(I-2P)(I-P)\theta^{\dagger}=(I-P)\theta^{\dagger}.$ (15) $$

Therefore, we have

$$ $f(\theta+\theta^{\dagger})=f((\theta+(I-P)\theta^{\dagger})+\theta^{\prime})=f ((I-2P)(\theta+(I-P)\theta^{\dagger})+\theta^{\prime})=f((I-2P)\theta+\theta^{ \dagger}).$ (16) $$

This proves part (1).

Part (2). By definition,

$$ $\displaystyle f^{\prime}(\theta-\theta^{\dagger}+\theta^{\prime})$ $\displaystyle=f(\theta+\theta^{\prime})$ (17) $\displaystyle=f((I-2P)\theta+\theta^{\prime})$ (18) $\displaystyle=f^{\prime}((I-2P)\theta-\theta^{\dagger}+\theta^{\prime}).$ (19) $$

This completes the proof.
∎

###### Proof.

### A.2 Proof of Proposition 2

###### Proof.

By (2), $g(x,\theta)$ simplifies to a kernel model

$$ $g(x,\theta)=\nabla_{\theta_{0}}f(x,\theta_{0})(I-P)\theta.$ (20) $$

Let us consider the squared loss $\ell(\theta)=\sum_{x}||y(x)-g(x,\theta)||^{2}$ and denote $A:=\sum_{x}(I-P)\nabla_{\theta_{0}}f(x,\theta_{0})^{T}\nabla_{\theta_{0}}f(x,
\theta_{0})(I-P)$, $b:=(I-P)\sum_{x}\nabla_{\theta_{0}}f(x,\theta_{0})^{T}y(x)$. Assuming the learning rate to be $\eta$, the GD reads

$$ $\theta^{t+1}=\theta^{t}-2\eta(A\theta^{t}-b),$ (21) $$

where $\theta^{0}=0$. If

$$ $\eta<\frac{1}{2\lambda_{max}(A)},$ (22) $$

GD converges to

$$ $\displaystyle\theta^{*}$ $\displaystyle=\lim_{t\to\infty}\sum_{k=0}^{t}(I-2\eta A)^{k}*2\eta b$ (23) $\displaystyle=A^{+}b,$ $$

which is the well-known least square solution.
∎

###### Proof.

### A.3 Proof of Proposition 3

###### Proof.

According to Ref. [, Theorem 4], we have

$$ $P\nabla_{\theta}\ell(x,\theta_{0})=0.$ (24) $$

Therefore, after one step of GD or SGD, we still have $P\theta_{1}=0$. By induction, we have $P\theta_{t}=0$.

Finally, suppose that $\{a_{i}\}_{i=1}^{d-{\rm rank}(P)}$ forms a basis of $\text{ker}P$, and define $f^{\prime}(x,\theta^{\prime}):=f(x,\sum_{i=1}^{d-{\rm rank}(P)}\theta_{i}^{
\prime}a_{i})$ for ${\rm dim}(\theta^{\prime})=d-{\rm rank}(P)$. By choosing $\theta^{\prime}_{i}=\theta^{T}a_{i}$, we have $f^{\prime}(x,\theta_{t}^{\prime})=f(x,\theta_{t})$.
∎

###### Proof.

### A.4 Lemmas

###### Lemma 1 .

Let $x\in\mathbb{R}^{d}$ and $P$ be a projection matrix. Let $f(x)$ be a scalar function that satisfies

$$ $f(x+x^{\prime})=f((I-2P)x+x^{\prime})+c^{T}Px,$ (25) $$

where $c$ is a constant vector. Then, there exists a unique function $g(x)$ such that

- 1.
$g(x)$ has the $x^{\prime}$-centric $P$-symmetry,
- 2.
and $f(x)=g(x)+\frac{1}{2}c_{0}^{T}Px$.

###### Proof.

(a) Existence. $f(x)=g(x)+\frac{1}{2}c_{0}^{T}x$. Let us suppose $g(x)$ is not $x^{\prime}$-centric $P$-symmetry. Then, there exists $x$ such that

$$ $g(x+x^{\prime})-g((I-2P)x+x^{\prime})=\Delta\neq 0.$ (26) $$

Then, by definition, we have that

$$ $\displaystyle c_{0}^{T}Px$ $\displaystyle=f(x+x^{\prime})+f((I-2P)x+x^{\prime})$ (27) $\displaystyle=g(x+x^{\prime})-\frac{1}{2}c_{0}^{T}P(x+x^{\prime})-g((I-2P)x+x^ {\prime})-c_{0}^{T}P(x+x^{\prime})$ (28) $\displaystyle=\Delta+\frac{1}{2}c_{0}^{T}P(x+x^{\prime})-\frac{1}{2}c_{0}^{T}P ((I-2P)x+x^{\prime})$ (29) $\displaystyle=\Delta+c_{0}^{T}Px.$ (30) $$

This is a contradiction. Therefore, there must exist $g(x)$ that satisfies the lemma statement.

(b) Uniqueness. Simply note that the expression of $g$ is uniquely given by(^6^66Alternatively, note that $c^{T}x$ is odd and that $g(x)$ can be shifted by a constant to be an even function. The uniqueness follows directly from the fact that every function can be uniquely factorized into an odd function and an even function.)

$$ $g(x)=f(x+x^{\prime})-f((I-2P)x+x^{\prime}).$ (31) $$

∎

###### Lemma 1 .

###### Proof.

### A.5 Proof of Theorem 1

###### Proof.

We prove by contradiction. Let us suppose there exists such pair, $(\theta^{\prime},P)$.
By definition, we have that

$$ $\ell_{\rm r}(\theta+\theta^{\prime})=\ell(\theta+\theta^{\prime}+\theta_{0})+ \gamma\|\theta+\theta^{\prime}\|^{2}.$ (32) $$

By assumption, we have that

$$ $\ell((I-2P)\theta+\theta^{\prime}+\theta_{0})+\gamma\|(I-2P)\theta+\theta^{ \prime}\|^{2}=\ell(\theta+\theta^{\prime}+\theta_{0})+\gamma\|\theta+\theta^{ \prime}\|^{2},$ (33) $$

and, so, for all $\theta$,

$$ $\ell((I-2P)\theta+\theta^{\prime}+\theta_{0})=\ell(\theta+\theta^{\prime}+ \theta_{0})+4\gamma\theta^{T}P\theta^{\prime}.$ (34) $$

There are two cases: (1) $P\theta^{\prime}=0$ and (2) $P\theta^{\prime}\neq 0$.

For case (1), we have that $\ell((I-2P)\theta+\theta^{\prime}+\theta_{0})=\ell(\theta+\theta^{\prime}+
\theta_{0})$, but this can only happen if the original loss $\ell$ has the $(\theta^{\prime}+\theta_{0})$-centric $P$-symmetry. By Property 1, this implies that

$$ $\theta^{\prime}+\theta_{0}=\theta^{\dagger}_{i}$ (35) $$

for some $i$. Applying $P$ on both sides, we obtain that

$$ $P\theta_{0}=P\theta^{\dagger}_{i}.$ (36) $$

But, $\theta_{0}$ is a random variable with a full-rank covariance while the set $\{P\theta^{\dagger}_{i}\}$ has measure zero in the real space, and so this equality holds with probability zero.

For case (2), it follows from Lemma 1 that for a fixed $x$ and $\theta_{0}$, $\ell(\theta)$ can be uniquely decomposed in the following form

$$ $\ell(\theta)=g(\theta)-2\gamma\theta^{T}P\theta^{\prime},$ (37) $$

where $g(\theta)$ has the $\theta^{\prime}+\theta_{0}$-centric $P$-symmetry.

Let $c_{0}=2\gamma P\theta^{\prime}$ and $\tilde{\theta}=P(\theta^{\prime}+\theta_{0})$. Then $\ell(\theta)+c_{0}\theta$ has the $\tilde{\theta}$-centric $P$ symmetry. We also have $\tilde{\theta}-\frac{c_{0}}{2\gamma}=P\theta_{0}$. According to Assumption 1, there are only countable many such $\{c_{0},\tilde{\theta}\}$ pairs. However, $P\theta_{0}$ is a standard Gaussian random variable, which leads to a contradiction with probability $1$.
∎

###### Remark .

It is easy to see that Assumption 1 could be slightly relaxed. We only require $\{\tilde{\theta}-\frac{c_{0}}{2\gamma}\}$ to have a zero measure, which is also a necessary and sufficient condition.

###### Proof.

###### Remark .

### A.6 Proof of Theorem 2

###### Proof.

We prove by contradiction. Let us suppose there exists such pair, $(\theta^{\prime},P)$.
By definition, we have that

$$ $\ell_{\rm ar}(\theta+\theta^{\prime})=\ell(\theta+\theta^{\prime}+\theta_{0})+ \gamma\|\theta+\theta^{\prime}\|_{D}^{2}.$ (38) $$

By assumption, we have that

$$ $\ell((I-2P)\theta+\theta^{\prime}+\theta_{0})+\gamma\|(I-2P)\theta+\theta^{ \prime}\|_{D}^{2}=\ell(\theta+\theta^{\prime}+\theta_{0})+\gamma\|\theta+ \theta^{\prime}\|_{D}^{2},$ (39) $$

and, so, for all $\theta$,

$$ $\ell((I-2P)\theta+\theta^{\prime}+\theta_{0})=\ell(\theta+\theta^{\prime}+ \theta_{0})+4\theta^{T}PD((I-P)\theta+\theta^{\prime}).$ (40) $$

There are two cases: (1) $PD((I-P)\theta+\theta^{\prime})=0$ and (2) $PD((I-P)\theta+\theta^{\prime})\neq 0$.

Like before, there are two cases. For case (2), the proof is identical, and so we omit it. For case (1), it must be the case that for some $P$, and $\theta^{\prime}$

$$ $\ell((I-2P)\theta+\theta^{\prime}+\theta_{0})=\ell(\theta+\theta^{\prime}+ \theta_{0}).$ (41) $$

This is possible if and only if $P(\theta^{\prime}+\theta_{0})=\theta^{\dagger}_{i}$ for some $i$ and $P=P_{i}$ for the corresponding projection matrix. However, because $P\theta^{\prime}=0$, this requires that

$$ $P(\theta^{\prime}+\theta_{0})=\theta_{i}^{\dagger}.$ (42) $$

By the definition of the reflection symmetry, we have that

$$ $P\theta^{\prime}=\theta^{\prime}.$ (43) $$

This means that

$$ $\theta^{\prime}=\theta_{i}^{\dagger}-P\theta_{0}.$ (44) $$

At the same time, we have

$$ $PD((I-P)\theta+\theta^{\prime})=0,$ (45) $$

which implies that

$$ $PD(I-P)\theta=-PD\theta^{\prime}.$ (46) $$

Because the right hand side is a constant that only depends on $\theta$. This can only happen if both sides are zero, which is achieved if:

$$ $PD(I-P)=0,$ (47) $$

and

$$ $\theta_{i}^{\dagger}=P\theta_{0}.$ (48) $$

The first condition implies that

$$ $PD=PDP=D^{T}P^{T}=DP,$ (49) $$

which implies that $P$ and $D$ must share the eigenvectors because they commute. Noting that

$$ $P\theta_{i}^{\dagger}=\theta_{i}^{\dagger},$ (50) $$

we obtain that $\theta_{i}^{\dagger}$ is an eigenvector of $P$ and so $\theta_{i}^{\dagger}$ is an eigenvector of $D$, but $D$ is diagonal and with nonidentical diagonal entries, $\theta_{i}^{\dagger}$ much then be a one-hot vector, and $P$ must also be diagonal and consists of values of $1$ and 0 0 in the diagonal entries.

∎

###### Proof.

### A.7 Proof of Theorem 3

###### Proof.

First of all, note that shifting parameters of $\ell$ is the same as shifting the parameters of the weight decay. Therefore, for any local minimum $\theta^{\prime}$,

$$ $\ell_{\rm ar}(\theta^{\prime})=\ell(\theta)+\gamma\|\theta-\theta_{0}\|^{2},$ (51) $$

where $\theta=\theta^{\prime}+\theta_{0}$.

Therefore,

$$ $\displaystyle{\ell_{\rm ar}(\theta^{\prime}+\theta^{*})-\ell_{\rm ar}((I-2P) \theta^{\prime}+\theta^{*})}$ (52) $\displaystyle=\ell(\theta+\theta^{*})+\gamma\|\theta+\theta^{*}-\theta_{0}\|_{ D}^{2}-\ell((I-2P)\theta+\theta^{*})-\gamma\|(I-2P)\theta+\theta^{*}-\theta_{0 }\|_{D}^{2}$ (53) $\displaystyle=\gamma\|\theta+\theta^{*}-\theta_{0}\|_{D}^{2}-\gamma\|(I-2P) \theta+\theta^{*}-\theta_{0}\|_{D}^{2}$ (54) $\displaystyle=\gamma(z_{\perp}^{T}(D-I)z_{\parallel}+z_{\perp}^{T}D(\theta^{*} -\theta_{0})),$ (55) $$

where we have used the definition of reflection symmetry in the third line. In the fourth line, we have defined $z_{\perp}=P\theta$ and $z_{\parallel}=(I-P)\theta$. Thus,

$$ $\|\theta\|^{2}=\|z_{\perp}\|^{2}+\|z_{\parallel}\|^{2}=\Theta(1).$ (56) $$

Thus, we have that

$$ $\displaystyle\Theta({\ell_{\rm ar}(\theta^{\prime}+\theta^{*})-\ell_{\rm ar}(( I-2P)\theta^{\prime}+\theta^{*})})$ $\displaystyle=\gamma\Theta(z_{\perp}^{T}(D-I)z_{\parallel})+\Theta(z_{\perp}^{ T}D(\theta^{*}-\theta_{0}))$ (57) $\displaystyle=\gamma\Theta(\sigma_{D}\|z_{\perp}\|)+\Omega(\sigma_{0}\|z_{ \perp}\|),$ (58) $$

where we have used the fact that each element of $\theta^{*}-\theta_{0}$ is $\Omega(\sigma_{0})$ because $\theta^{*}$ is an arbitrary constant and $\theta_{0}\sim\mathcal{N}(0,\sigma^{2})$. By the assumption that $\sigma_{D}=o(\sigma_{0})$, we obtain the desired relation

$$ ${\ell_{\rm ar}(\theta^{\prime}+\theta^{*})-\ell_{\rm ar}((I-2P)\theta^{\prime} +\theta^{*})}=\Omega(\gamma\sigma_{0})\|z_{\perp}\|.$ (59) $$

This finishes the proof.
∎

###### Proof.

### A.8 Proof of Theorem 4

###### Proof.

First of all,

$$ $\displaystyle\Pr(\min_{i}|\Delta_{i}|>\epsilon)$ $\displaystyle=\Pr(|\Delta_{1}|>\epsilon\land...\land|\Delta_{N}|>\epsilon)$ (60) $\displaystyle\geq\max(0,\sum_{i}^{N}\Pr(|\Delta_{i}|>\epsilon)-N+1),$ (61) $$

where we have applied the Frechet inequality in the second line.

The sum $\sum_{i}^{N}\Pr(|\Delta_{i}|>\epsilon)$ can also be lower bounded if each $\Delta_{i}$ is a Gaussian variable with variance $\sigma_{i}$:

$$ $\displaystyle\sum_{i}^{N}\Pr(|\Delta_{i}|>\epsilon)$ $\displaystyle\geq\sum_{i}^{N}(1-\frac{2\epsilon}{\sqrt{2\pi\sigma_{i}^{2}}})$ (62) $\displaystyle\geq N-\frac{2\epsilon N}{\min_{i}\sqrt{2\pi\sigma_{i}^{2}}}.$ (63) $$

Thus, for $\Pr(\min_{i}|\Delta_{i}|>\epsilon)$ to be larger than $1-\delta$, we must have

$$ $\min_{i}\sigma_{i}\geq\frac{2\epsilon N}{\sqrt{2\pi}(1-\delta)}.$ (64) $$

Now, we show that $\Delta_{i}$-s are indeed Gaussian variables. From the previous proof, it is easy to see that for a unit vector $n_{i}$,

$$ $\Delta_{i}=\gamma n_{i}^{T}(\theta^{*}-\theta_{0})+o(\gamma\sigma_{0}).$ (65) $$

Therefore, $\Delta_{i}$ is a Gaussian variable with standard deviation $\gamma\sigma_{0}$. Thus, $\min_{i}\sigma_{i}=\gamma\sigma_{0}$. Thus, we have obtained the desired result

$$ $\gamma\sigma_{0}\geq\frac{2\epsilon N}{\sqrt{2\pi}(1-\delta)}.$ (66) $$

∎

###### Proof.

### A.9 Proof of Theorem 5

Before we start proving Theorem 5, we introduce a definition that facilitates the proof.

###### Definition 2 .

(Symmetry reduction.) We say that removing a symmetry from group $G_{1}$ reduces to removing the symmetry due to group $G_{2}$ if for any vector $n$,

$$ $\|(I-\overline{G_{2}})n\|\leq\|(I-\overline{G_{1}})n\|.$ (67) $$

Now, we are ready to prove the main theorem.

###### Proof.

We first show $(I-\overline{V})^{T}\nabla_{\theta}\ell(\theta)=0$. For any $g\in V$ and $z\in\mathbb{R}$, we have

$$ $\ell(\theta+zn)=\ell(g(\theta+zn)),$ (68) $$

where $n$ is an arbitrary unit vector.
Taking the derivative with respect to $z$, and recalling that
$g\theta=\theta$, we have

$$ $(gn)^{T}\nabla_{\theta}\ell(\theta)=n^{T}\nabla_{\theta}\ell(\theta).$ (69) $$

Accordingly, we have

$$ $(Vn)^{T}\nabla_{\theta}\ell(\theta):=\frac{1}{|V|}\sum_{g\in V}(gn)^{T}\nabla_ {\theta}\ell(\theta)=n^{T}\nabla_{\theta}\ell(\theta).$ (70) $$

Due to the arbitrary choice of $n$, we have $(I-\overline{V})^{T}\nabla_{\theta}\ell(\theta)=0$.

Therefore,

$$ $\displaystyle(I-\overline{V})^{T}\nabla_{\theta}\ell_{\rm ar}(\theta)$ $\displaystyle=\gamma(I-\overline{V})^{T}\nabla_{\theta}\|\theta-\theta_{0}\|_{ D}^{2}$ (71) $\displaystyle=2\gamma(I-\overline{V})^{T}D(\theta-\theta_{0})$ (72) $\displaystyle=2\gamma(I-\overline{V})^{T}\theta_{0}+o(\gamma\sigma_{0}(1+ \sigma_{D})).$ (73) $$

As $\theta_{0}$ is a Gaussian vector with mean 0 0 and variance $\sigma_{0}^{2}$, $||(I-\overline{V})^{T}\theta_{0}||$ is a Gaussian variable with mean 0 0 and variance $||I-\overline{V}||^{2}\sigma_{0}^{2}=\Omega({\rm rank}(I-\overline{V})\sigma_{
0}^{2})$, which gives
$\|(I-\overline{V})\nabla_{\theta}\ell\|=\Omega(\gamma\sigma_{0}\sqrt{{\rm rank
}(I-\overline{V})}).$

Now, we prove part (2) of the theorem. Note that if $V\lhd U$ and if $\theta_{0}\in\ker\overline{V}$, then $\theta_{0}\in\ker\overline{U}$. This means that for any group $U$ such that $V\lhd U$ and any vector $\theta_{0}$

$$ $\|(I-\overline{V})\theta_{0}\|<\|(I-\overline{U})\theta_{0}\|.$ (74) $$

This means that to remove the symmetry from a large group $U$, it suffices to remove the symmetry from one of its minimal subgroups. Thus, let $M_{G}$ denote the set of minimal subgroups of the group $G$, we have

$$ $\min_{V\lhd G}\|(I-\overline{V})\theta_{0}\|\geq\min_{V\lhd M_{G}}\|(I- \overline{V})\theta_{0}\|.$ (75) $$

The number of minimal subgroups is strictly upper bounded by the number of elements of the group because all minimal subgroups are only trivially intersect each other. This follows from the fact that the intersection of groups must be a subgroup, which can only be the identity for two different minimal subgroups. Therefore, the number of minimal subgroups cannot exceed the number of elements of the group. This finishes the second part of the theorem.

For the third part, we show that the symmetry broken subspace of any subgroup contains the symmetry broken subspace of a group generated by one of the generators and so it suffices to only remove the symmetries due to the subgroups generated by each generator.
Let us introduce the following notation for a matrix representation $z$ of a group element:

$$ $\overline{z}=\frac{1}{\rm ord(z)}\sum_{i=1}^{\rm ord(z)}z^{i},$ (76) $$

where ${\rm ord}(z)$ denotes the order of $z$. This is equivalent to the symmetry projection matrix of the subgroup generated by $z$.

Now, let $G$ be abelian. Then, both $U$ and $V$ are abelian. Let us denote by $\Gamma(U)=\{z_{i}\}$ the mininal generating set of $U$. Suppose that for all $n\neq 0$ such that $n\in{\rm im}(I-\overline{V})$, we must have $n\notin{\rm im}(I-\overline{z_{j}})$ for all $j$. This means that

$$ $n\notin\bigcup_{j}{\rm im}(I-\overline{z_{j}}).$ (77) $$

Or, equivalently,

$$ $n\in\bigcap_{j}{\rm im}(\overline{z_{j}}).$ (78) $$

However, the space $\bigcap_{j}{\rm im}(\overline{z_{j}})\subseteq{\rm im}(\overline{V})$ because $V$ is a subgroup of $U$, which is generated by $z_{1},\cdots,z_{m}$. To see this, let $n\in{\rm im}(\overline{z_{j}})$ for all $j$, then,

$$ $z_{j}n=n$ (79) $$

for all $j$. Now, let $v=\prod_{i}z_{i}^{d_{i}(v)}\in V$, we have

$$ $(I-\overline{V})n=(I-\sum_{v}\prod_{i}z_{i}^{d_{i}(v)})n=0$ (80) $$

This means that $n$ is in both ${\rm im}(\overline{V})$ and ${\rm im}(I-\overline{V})$, which is possible only if $n=0$ – a contradiction. Therefore, as long as $I-\bar{V}$ is not rank 0 0, it must share a common subspace with one of the $I-\overline{z_{j}}$, and so removing the symmetry from any subgroup $V$ of $U$ can be reduced to removing the symmetry from the cyclic group generated by one of its generators from the minimal generating set.(^7^77This holds true even if $V$ is a subset of $\langle z_{j}\rangle$.)

Therefore, we have proved that removing symmetries due to any subgroup of $U$ can be reduced to removing the symmetry from a (proper or trivial) subgroup of each of the cyclic decompositions of the group $U$, each of which is generated by a minimal generator of $U$. By the fundamental theorem of finite abelian groups, each of these groups is of order $p^{k}$ for some prime number $p$. Because each of these groups is cyclic, it contains exactly $k$ nontrivial subgroups. Taken together, this means that if $|U|=p_{1}^{k_{1}}...p_{m}^{k_{m}}$, we only have to remove symmetries from at most

$$ $\sum_{i}^{m}k_{i}=\log|U|$ (81) $$

many subgroups. This completes part (3).

For part (4), we denote $\Delta_{i}:=(I-\overline{V_{i}})^{T}\nabla_{\theta}\ell_{\rm ar}(\theta)$ for $i=1,\cdots,|\Gamma(U)|$. According to (73), $\Delta_{i}$ is approximately a Gaussian variable with zero mean and variance $\gamma^{2}{\rm rank}(I-\overline{V})\sigma_{0}^{2}$. Therefore,

$$ $\displaystyle\sum_{i}^{|\Gamma(U)|}\Pr(|\Delta_{i}|>\epsilon)$ $\displaystyle\geq|\Gamma(U)|-\frac{2\epsilon|\Gamma(U)|}{\min_{i}\sqrt{2\pi \gamma^{2}{\rm rank}(I-\overline{V})\sigma_{0}^{2}}}.$ (82) $$

For $\Pr(\min_{i}|\Delta_{i}|>\epsilon)$ to be larger than $1-\delta$, we must have

$$ $\gamma\sigma_{0}\geq\frac{2\epsilon|\Gamma(U)|}{\sqrt{2\pi\ {\rm rank}(I- \overline{V})}(1-\delta)}.$ (83) $$

∎

###### Definition 2 .

###### Proof.

## Appendix B Additional Experiments and Experimental Detail

### B.1 Teacher-student Scenario

This section gives some additional details and additional experiments in the teacher-student scenario in Figure 2. Specifically, we implement a two-layer network with tanh activation, $300$ hidden units, and different input units. The network outputs a ten-dimensional vector corresponding to ten different classes. We then randomly generate such a network as the teacher model, $10000$ standard Gaussian samples as the training set, and $1000$ standard Gaussian samples. For both the syre and the vanilla model, we choose the Adam optimizer, learning rate $0.01$, and weight decay $0.01$.

As additional experiments, we also measure the influence of noisy labels and noisy input on the rank of the model in Figure 10. For the label noise, we randomly change $0\%$ to $80\%$ of the labels, and for the input noise, we add a Gaussian noise to the input with standard deviation 0 0 to $1.6$. Figure 10 suggests that the rank of the vanilla model decreases in the face of noisy labels and increases in the face of noisy input, perhaps because the latter can be regarded as data augmentation. The syre model, however, is not affected.

Figure: Figure 10: For the same setting as Figure 2, the rank of the covariance matrix decreases with the label noise and increases with the input noise. We set eigenvalues smaller than $10^{-3}$ to zero.
Refer to caption: extracted/5817815/figures/label_noise.png

### B.2 Supervised Learning

This section presents some additional experiments for Section 6.3. Figure 11 gives the eigenvalue distribution of the networks in Fig.5, which further supports the claim that the vanilla network leads to a low-rank solution.
In all the experiments above and in Section 6.3, we use a four-layer FCN with $300$ neurons in each layer trained on the MNIST dataset with batch size $64$.

Figure: Figure 11: The spectrum of the covariance matrix of the vanilla model (Left) and the syre model (Right) for $\gamma=0.01$ and $\alpha=1$. Clearly, the vanilla model learns a low-rank solution.
Refer to caption: extracted/5817815/figures/unbiased.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. It would be beneficial to see further exploration into the limits of the method, especially in more complex or less controlled environments than those tested. Also, maybe trying some transformer architecture based methods for SSL might make the architecture agnostic point stronger? 

2. Could the authors elaborate on any potential computational overhead or scalability issues when implementing syre in larger or more complex models?
soundness: 3
strengths: 1. The proposed method is theoretically sound. Authors have rigorously proved many useful results like Proposition 3 which shows symmetry reduces parameter dimension.
2. Their proposed solution is generalizable across different architectures and does not require detailed knowledge of the model’s symmetries.
3. The empirical evaluations demonstrate significant improvements in model performance and capacity, particularly in challenging scenarios like continual learning and self-supervised learning setups.
summary: This paper presents a novel approach for addressing the limitations posed by symmetries in neural network training, which often lead to low-capacity model states. The authors propose a method called "syre", which can effectively remove almost all types of symmetries from neural networks, thereby enhancing model expressivity and performance across various training scenarios. The method is both model and symmetry group agnostic and straightforward to implement, requiring only minimal changes to existing training pipelines.
weaknesses: 1. Some of the theoretical explanations are dense and might be challenging for readers not familiar with the detailed aspects of symmetry in neural networks. Having decent knowledge of symmetries in neural networks I still struggled a bit with some sections like Section 5. Summarizing early on what the main theoretical result of each section might make it easier for reader to appreciate the contributions of the paper better. 

2. The limitations of the method are not thoroughly discussed, which could be beneficial for practitioners to know when to use this technique with caution. 

3. The formatting of the paper seems off. All the headings are pretty close to the previous paragraph hurting the readability.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: - How does your syre do on the (Lim 2024) LMC experiments?
- Do you have any (even at a very high level) design rules for how a training-algorithm designer may usefully modulate the symmetries in a model?
soundness: 3
strengths: - As noted, provide some plausible and interesting explanations for some mechanisms for how symmetry may reduce capacity that nontrivially build on (Zivin 2024).
- In some settings and experiments, their algorithm (syre) seems to usefully outperform baselines/another symmetry removal method?
summary: Main contributions are:
- Builds on (Ziyin 2024) to introduce two ways that symmetry may reduce model capacity— a) that in the lazy training limit, the P-symmetry ‘masks’ certain features around a symmetric solution and b) the parameter dimension is reduced by the rank of P when around a symmetric solution. 
- Introduces a straightforward approach to eliminate these symmetric solutions by both fixing a random bias on the neurons (to handle finite symmetries) and also modifying weight decay to use a norm with respect to a positive diagonal matrix D (to handle infinite symmetries, e.g. like those that appear in the residual stream of a transformer). Also introduce a measure to quantify the degree to which a symmetry is broken. 
- Experiments that measure the extent of symmetry removal for some linear regression set-ups and some vision tasks, and continual learning in an RL setting.
weaknesses: Experimental shortcomings
- The authors make strong claims around the ability of symmetry-removal to improve performance for a) SSL, b) a supervised continual learning problem, and c) an RL continual learning problem. They do not implement proper baseline comparisons for any of these settings nor go beyond toy experiments. If they want to claim that (some degree of) symmetry removal is helpful for training models in practical settings, they need to either show that a) symmetry removal is reasonably competitive in mildly (compute/FLOP-controlled or at least aware) realistic settings or b) symmetry removal explains some % of the success of methods that do well in these settings. Aside from performance gains, benchmarking their method against the (Lim 2024) methods on linear mode connectivity would also help better validate their set-up. Likewise, benchmarking on other approaches on symmetry-removal.
- On the other hand, the authors only very loosely test their explanations in Prop. 2/3. Presumably, if this explanation is correct, shouldn’t models with more/less symmetries be less/more stuck in neuron-collapsed solutions (e.g. in the spirit of the Zivin 2024 experiments)? Fig. 1 only shows one case. If this claim around the science of models is to be taken as a main contribution of the paper, more experimental evidence of the propositions is needed.


Miscellaneous
- Typo on line 65-66: “$\|\theta\|_D^2:=w^T D w$ is the norm of $w$ with respect to $D$” should be “$|w|_D^2:=w^T D w$ is the norm of $w$ with respect to $D”


Ziyin, Liu. "Symmetry Induces Structure and Constraint of Learning." Forty-first International Conference on Machine Learning.
Lim, Derek, et al. "The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof." arXiv preprint arXiv:2405.20231 (2024).

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Those are good questions. Here are those formatted plus a few additional ones I think would be helpful:

1. **Fixed Points vs Symmetries**:
   - Do you mean fixed points (invariant sets) when discussing "symmetric points" where $P\theta_0 = 0$? If so, consider revising terminology to avoid confusion with symmetry orbits.
   - Why focus on fixed points rather than the full symmetry orbits which characterize the degeneracies in the loss landscape?

2. **Gradient Analysis**:
   - Can you clarify the argument around equation (2) about vanishing gradients? How does this differ from the natural vanishing of gradients along symmetry orbits?
   - In the expansion around fixed points, do you consider gradients in directions orthogonal to the symmetry?

3. **Initialization and Fixed Points**:
   - For a simple network $F(x) = UVx$, the origin $(U,V)=(0,0)$ is a fixed point, but why is this relevant when standard initialization (Glorot, etc.) avoids such points?
   - Do your arguments about trapping at fixed points apply to more realistic initialization schemes?

4. **Regularization Effects**:
   - Can you provide ablation studies comparing your method with standard weight decay?
   - For the D-norm modification, how does making weight decay ellipsoidal fundamentally differ from standard weight decay in terms of symmetry breaking?

5. **Theoretical Claims**:
   - The condition $P\theta' = \theta'$ seems unnecessarily restrictive. Can your results be generalized without this condition?
   - For continuous symmetries (like $GL(h)$ in linear networks), can you make precise what subset of symmetries your method breaks?

6. **Empirical Validation**:
   - Could you provide standard weight decay baselines for all experiments?
   - In scenarios where your method outperforms standard weight decay (like continual learning), what mechanism explains this improvement?

Would you like to add any other questions?
soundness: 3
strengths: Here are the main strengths of the paper:

1. Addresses an important practical problem of symmetry-induced model collapse that affects multiple domains in deep learning (VAEs, continual learning, self-supervised learning).

2. Proposes a remarkably simple solution that requires minimal code changes and no architectural modifications, making it easily adoptable in practice.

3. Shows promising results in continual learning scenarios, demonstrating improved plasticity and performance maintenance over time compared to standard approaches.

4. Makes an interesting connection between symmetries and model collapse, providing a framework for thinking about how symmetries might limit model expressivity.
summary: This paper addresses the problem of symmetries in neural network loss landscapes and their role in causing models to get trapped in low-capacity states. The authors first prove that symmetries in the loss function can lead to reduced model capacity through two mechanisms: feature masking and parameter dimension reduction. They then propose SYRE, a simple modification to neural network training that aims to remove symmetries by adding a random bias to the parameters along with weight decay. For cases with uncountably many symmetries, they propose an advanced version that additionally uses a random diagonal matrix in the weight decay term. The method is model-agnostic and requires no knowledge of the underlying symmetries. The authors demonstrate their method's effectiveness across several applications where symmetry-induced capacity reduction is a concern, including supervised learning, variational autoencoders, self-supervised learning, and continual learning. They particularly show improvements in scenarios where models typically suffer from various forms of collapse, such as posterior collapse in VAEs and loss of plasticity in continual learning.
weaknesses: 1. **Conceptual confusion about symmetries**: 
   - The paper discusses "removing symmetries" but primarily analyzes fixed points where $P\theta_0 = 0$
   - This misses the key aspect of symmetries - their group orbits and the associated degeneracies
   - In the simple case of a two-layer linear network $F(x) = UVx$, the important symmetry structure lies in the $GL(h)$ orbits $(U,V) \to (Ug, g^{-1}V)$, not in fixed points

2. **Overly restrictive theoretical framework**:
   - The condition $P\theta' = \theta'$ for $(\theta', P)$-reflection symmetry is artificially restrictive
   - Simple examples show valid reflections that violate this condition
   - For instance, in 2D with $\theta' = (0,1)$ and reflection in x-direction $P = ((1,0),(0,0))$, we have $P\theta' \neq \theta'$ yet this is a perfectly valid reflection symmetry

3. **Questionable arguments about gradient vanishing**:
   - The paper suggests symmetric points trap models due to vanishing gradients
   - However, in the UV example, fixed points along $U=V$ line have nonzero gradients except at origin and minima
   - Gradients naturally vanish along symmetry orbits (they're level sets of the loss), but this doesn't imply trapping as claimed

4. **Limited analysis of regularization effects**:
   - The D-norm modification merely changes spherical weight decay to ellipsoidal weight decay
   - The $\theta_0$ shift changes the center of the regularization
   - While these modifications may improve optimization, it's not clear they "remove symmetries" as claimed
   - In the UV example, these modifications may select preferred points along symmetry orbits (like weight decay does) but may not fundamentally break the symmetry structure

5. **Experimental limitations**:
   - Lack of systematic comparison with standard weight decay baseline
   - Only a few experiments (particularly continual learning) show clear improvements over weight decay
   - Results suggest their method might be a useful variant of weight decay, but not necessarily for the theoretical reasons provided

The paper's suggested regularization may indeed have advantages over standard weight decay in certain scenarios, but the theoretical arguments about symmetry breaking are not convincing, and the empirical validation needs stronger baselines to support the claims.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: - How does syre scale to large models like transformers? Have you tested its efficiency or computational impact on such architectures?
- Would an adaptive bias approach improve the method compared to using a static bias throughout training?
- How sensitive is syre to hyperparameter choices, and do you have heuristics to guide practitioners in tuning them?
soundness: 3
strengths: - The paper presents a novel approach to tackling symmetry-induced collapses in neural networks. The proposed method uses static random biases to remove symmetries, a simple yet innovative solution. The theoretical insights connect symmetry and expressivity in a new way, offering a significant advancement.
- The theoretical contributions are rigorous and well-supported with proofs. The authors also provide a range of empirical validations that align with their theoretical claims, making a strong case for the method’s effectiveness across various scenarios.
- The paper is clearly structured, with a logical flow from problem setup to solution. The core concepts are well-explained.
- The method's simplicity and general applicability make it impactful. By addressing a fundamental problem in deep learning training, syre has potential benefits for diverse architectures and tasks, opening new research avenues into model expressivity and optimization.
summary: This paper addresses the problem of low-capacity states, often referred to as "collapse," that arise when neural network loss functions contain inherent symmetries. The authors prove two mechanisms through which symmetries reduce model expressivity, leading to these collapsed states and propose a method called syre to eliminate such symmetries. The syre algorithm introduces static random biases alongside weight decay, providing a simple yet effective means to remove almost all symmetry-induced constraints without requiring knowledge of the specific symmetries. The method is claimed to be model-agnostic, applicable to a broad range of architectures, and effective in both theoretical and practical contexts. Empirical results support the effectiveness of the proposed method in diverse settings, including supervised learning, self-supervised learning, and continual learning.
weaknesses: - Experiments are limited to small- or medium-scale models. The method’s applicability to real-world, large-scale tasks is insufficiently demonstrated. Benchmarks on high-impact datasets, such as ImageNet, and large-scale architectures, like transformers, would add credibility.
- The impact of completely removing symmetries on generalization is not well-discussed. Symmetry sometimes acts as a regularizer, and the absence of analysis on this potential downside is a gap.

Meta-review: **Summary** This paper studies the role of neural network parameter symmetries on the phenomenon of collapse in which the model is trapped in a low-capacity state during training.  The authors provide theory explaining how symmetry can lead to reduced capacity.  That propose a model-agnostic method called syre which is a variant of weight decay which can provably break symmetry during training and empirically leads to improved performance in domains where collapse has been an issue.

**Strengths** This paper provides an interesting and insightful theory linking parameter symmetries to reduced model capacity. This theory leads to a novel method, syre, which is simple, highly general, lightweight and can avoid symmetry and collapse. The empirical evaluations show syre is effective in practice in a range of settings (including for transformers after the rebuttal). 

**Weaknesses** Reviewers initially had several criticisms, many of which were reduced by the author response.  Some reviewers considered the claims in the paper to be overstrong, but the authors reduced these claims in the revision.  Additionally, while syre is claimed to improve performance through symmetry reduction, the experiments in the paper hinged mainly on improved model performance, failing to demonstrate the exact mechanism.   The authors helped to address this by adding experiments showing syre can increase linear mode connectivity. 

**Conclusion** The reviewers and the AC have a consensus that this paper provides an insightful theory connecting parameter symmetry to reduced model capacity and an effective method to mitigate the problem.  While there are potentially other experiments which can be performed with this method, I believe accepting the paper will help encourage that follow up work.

Decision: Accept (Poster)