Remove Symmetries to Control Model Expressivity and Improve Optimization

Authors: Liu Ziyin, Yizhou Xu, Isaac L. Chuang
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: model capacity, symmetry, optimization

Abstract:
When symmetry is present in the loss function, the model is likely to be trapped in a low-capacity state that is sometimes known as a ``collapse." Being trapped in these low-capacity states can be a major obstacle to training across many scenarios where deep learning technology is applied. We first prove two concrete mechanisms through which symmetries lead to reduced capacities and ignored features during training and inference. We then propose a simple and theoretically justified algorithm, \textit{syre}, to remove almost all symmetry-induced low-capacity states in neural networks. When this type of entrapment is especially a concern, removing symmetries with the proposed method is shown to correlate well with improved optimization or performance. A remarkable merit of the proposed method is that it is model-agnostic and does not require any knowledge of the symmetry.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730686032111

thread_type:
official_review

<thread_object>
creator: reviewer_akev

<content>
summary:
This paper addresses the problem of low-capacity states, often referred to as "collapse," that arise when neural network loss functions contain inherent symmetries. The authors prove two mechanisms through which symmetries reduce model expressivity, leading to these collapsed states and propose a method called syre to eliminate such symmetries. The syre algorithm introduces static random biases alongside weight decay, providing a simple yet effective means to remove almost all symmetry-induced constraints without requiring knowledge of the specific symmetries. The method is claimed to be model-agnostic, applicable to a broad range of architectures, and effective in both theoretical and practical contexts. Empirical results support the effectiveness of the proposed method in diverse settings, including supervised learning, self-supervised learning, and continual learning.

strengths:
- The paper presents a novel approach to tackling symmetry-induced collapses in neural networks. The proposed method uses static random biases to remove symmetries, a simple yet innovative solution. The theoretical insights connect symmetry and expressivity in a new way, offering a significant advancement.
- The theoretical contributions are rigorous and well-supported with proofs. The authors also provide a range of empirical validations that align with their theoretical claims, making a strong case for the method’s effectiveness across various scenarios.
- The paper is clearly structured, with a logical flow from problem setup to solution. The core concepts are well-explained.
- The method's simplicity and general applicability make it impactful. By addressing a fundamental problem in deep learning training, syre has potential benefits for diverse architectures and tasks, opening new research avenues into model expressivity and optimization.

weaknesses:
- Experiments are limited to small- or medium-scale models. The method’s applicability to real-world, large-scale tasks is insufficiently demonstrated. Benchmarks on high-impact datasets, such as ImageNet, and large-scale architectures, like transformers, would add credibility.
- The impact of completely removing symmetries on generalization is not well-discussed. Symmetry sometimes acts as a regularizer, and the absence of analysis on this potential downside is a gap.

questions:
- How does syre scale to large models like transformers? Have you tested its efficiency or computational impact on such architectures?
- Would an adaptive bias approach improve the method compared to using a static bias throughout training?
- How sensitive is syre to hyperparameter choices, and do you have heuristics to guide practitioners in tuning them?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730844911282

thread_type:
official_review

<thread_object>
creator: reviewer_pyez

<content>
summary:
This paper addresses the problem of symmetries in neural network loss landscapes and their role in causing models to get trapped in low-capacity states. The authors first prove that symmetries in the loss function can lead to reduced model capacity through two mechanisms: feature masking and parameter dimension reduction. They then propose SYRE, a simple modification to neural network training that aims to remove symmetries by adding a random bias to the parameters along with weight decay. For cases with uncountably many symmetries, they propose an advanced version that additionally uses a random diagonal matrix in the weight decay term. The method is model-agnostic and requires no knowledge of the underlying symmetries. The authors demonstrate their method's effectiveness across several applications where symmetry-induced capacity reduction is a concern, including supervised learning, variational autoencoders, self-supervised learning, and continual learning. They particularly show improvements in scenarios where models typically suffer from various forms of collapse, such as posterior collapse in VAEs and loss of plasticity in continual learning.

strengths:
Here are the main strengths of the paper:

1. Addresses an important practical problem of symmetry-induced model collapse that affects multiple domains in deep learning (VAEs, continual learning, self-supervised learning).

2. Proposes a remarkably simple solution that requires minimal code changes and no architectural modifications, making it easily adoptable in practice.

3. Shows promising results in continual learning scenarios, demonstrating improved plasticity and performance maintenance over time compared to standard approaches.

4. Makes an interesting connection between symmetries and model collapse, providing a framework for thinking about how symmetries might limit model expressivity.

weaknesses:
1. **Conceptual confusion about symmetries**: 
   - The paper discusses "removing symmetries" but primarily analyzes fixed points where $P\theta_0 = 0$
   - This misses the key aspect of symmetries - their group orbits and the associated degeneracies
   - In the simple case of a two-layer linear network $F(x) = UVx$, the important symmetry structure lies in the $GL(h)$ orbits $(U,V) \to (Ug, g^{-1}V)$, not in fixed points

2. **Overly restrictive theoretical framework**:
   - The condition $P\theta' = \theta'$ for $(\theta', P)$-reflection symmetry is artificially restrictive
   - Simple examples show valid reflections that violate this condition
   - For instance, in 2D with $\theta' = (0,1)$ and reflection in x-direction $P = ((1,0),(0,0))$, we have $P\theta' \neq \theta'$ yet this is a perfectly valid reflection symmetry

3. **Questionable arguments about gradient vanishing**:
   - The paper suggests symmetric points trap models due to vanishing gradients
   - However, in the UV example, fixed points along $U=V$ line have nonzero gradients except at origin and minima
   - Gradients naturally vanish along symmetry orbits (they're level sets of the loss), but this doesn't imply trapping as claimed

4. **Limited analysis of regularization effects**:
   - The D-norm modification merely changes spherical weight decay to ellipsoidal weight decay
   - The $\theta_0$ shift changes the center of the regularization
   - While these modifications may improve optimization, it's not clear they "remove symmetries" as claimed
   - In the UV example, these modifications may select preferred points along symmetry orbits (like weight decay does) but may not fundamentally break the symmetry structure

5. **Experimental limitations**:
   - Lack of systematic comparison with standard weight decay baseline
   - Only a few experiments (particularly continual learning) show clear improvements over weight decay
   - Results suggest their method might be a useful variant of weight decay, but not necessarily for the theoretical reasons provided

The paper's suggested regularization may indeed have advantages over standard weight decay in certain scenarios, but the theoretical arguments about symmetry breaking are not convincing, and the empirical validation needs stronger baselines to support the claims.

questions:
Those are good questions. Here are those formatted plus a few additional ones I think would be helpful:

1. **Fixed Points vs Symmetries**:
   - Do you mean fixed points (invariant sets) when discussing "symmetric points" where $P\theta_0 = 0$? If so, consider revising terminology to avoid confusion with symmetry orbits.
   - Why focus on fixed points rather than the full symmetry orbits which characterize the degeneracies in the loss landscape?

2. **Gradient Analysis**:
   - Can you clarify the argument around equation (2) about vanishing gradients? How does this differ from the natural vanishing of gradients along symmetry orbits?
   - In the expansion around fixed points, do you consider gradients in directions orthogonal to the symmetry?

3. **Initialization and Fixed Points**:
   - For a simple network $F(x) = UVx$, the origin $(U,V)=(0,0)$ is a fixed point, but why is this relevant when standard initialization (Glorot, etc.) avoids such points?
   - Do your arguments about trapping at fixed points apply to more realistic initialization schemes?

4. **Regularization Effects**:
   - Can you provide ablation studies comparing your method with standard weight decay?
   - For the D-norm modification, how does making weight decay ellipsoidal fundamentally differ from standard weight decay in terms of symmetry breaking?

5. **Theoretical Claims**:
   - The condition $P\theta' = \theta'$ seems unnecessarily restrictive. Can your results be generalized without this condition?
   - For continuous symmetries (like $GL(h)$ in linear networks), can you make precise what subset of symmetries your method breaks?

6. **Empirical Validation**:
   - Could you provide standard weight decay baselines for all experiments?
   - In scenarios where your method outperforms standard weight decay (like continual learning), what mechanism explains this improvement?

Would you like to add any other questions?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730940203598

thread_type:
official_review

<thread_object>
creator: reviewer_jsoa

<content>
summary:
Main contributions are:
- Builds on (Ziyin 2024) to introduce two ways that symmetry may reduce model capacity— a) that in the lazy training limit, the P-symmetry ‘masks’ certain features around a symmetric solution and b) the parameter dimension is reduced by the rank of P when around a symmetric solution. 
- Introduces a straightforward approach to eliminate these symmetric solutions by both fixing a random bias on the neurons (to handle finite symmetries) and also modifying weight decay to use a norm with respect to a positive diagonal matrix D (to handle infinite symmetries, e.g. like those that appear in the residual stream of a transformer). Also introduce a measure to quantify the degree to which a symmetry is broken. 
- Experiments that measure the extent of symmetry removal for some linear regression set-ups and some vision tasks, and continual learning in an RL setting.

strengths:
- As noted, provide some plausible and interesting explanations for some mechanisms for how symmetry may reduce capacity that nontrivially build on (Zivin 2024).
- In some settings and experiments, their algorithm (syre) seems to usefully outperform baselines/another symmetry removal method?

weaknesses:
Experimental shortcomings
- The authors make strong claims around the ability of symmetry-removal to improve performance for a) SSL, b) a supervised continual learning problem, and c) an RL continual learning problem. They do not implement proper baseline comparisons for any of these settings nor go beyond toy experiments. If they want to claim that (some degree of) symmetry removal is helpful for training models in practical settings, they need to either show that a) symmetry removal is reasonably competitive in mildly (compute/FLOP-controlled or at least aware) realistic settings or b) symmetry removal explains some % of the success of methods that do well in these settings. Aside from performance gains, benchmarking their method against the (Lim 2024) methods on linear mode connectivity would also help better validate their set-up. Likewise, benchmarking on other approaches on symmetry-removal.
- On the other hand, the authors only very loosely test their explanations in Prop. 2/3. Presumably, if this explanation is correct, shouldn’t models with more/less symmetries be less/more stuck in neuron-collapsed solutions (e.g. in the spirit of the Zivin 2024 experiments)? Fig. 1 only shows one case. If this claim around the science of models is to be taken as a main contribution of the paper, more experimental evidence of the propositions is needed.


Miscellaneous
- Typo on line 65-66: “$\|\theta\|_D^2:=w^T D w$ is the norm of $w$ with respect to $D$” should be “$|w|_D^2:=w^T D w$ is the norm of $w$ with respect to $D”


Ziyin, Liu. "Symmetry Induces Structure and Constraint of Learning." Forty-first International Conference on Machine Learning.
Lim, Derek, et al. "The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof." arXiv preprint arXiv:2405.20231 (2024).

questions:
- How does your syre do on the (Lim 2024) LMC experiments?
- Do you have any (even at a very high level) design rules for how a training-algorithm designer may usefully modulate the symmetries in a model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731384521756

thread_type:
official_review

<thread_object>
creator: reviewer_fj68

<content>
summary:
This paper presents a novel approach for addressing the limitations posed by symmetries in neural network training, which often lead to low-capacity model states. The authors propose a method called "syre", which can effectively remove almost all types of symmetries from neural networks, thereby enhancing model expressivity and performance across various training scenarios. The method is both model and symmetry group agnostic and straightforward to implement, requiring only minimal changes to existing training pipelines.

strengths:
1. The proposed method is theoretically sound. Authors have rigorously proved many useful results like Proposition 3 which shows symmetry reduces parameter dimension.
2. Their proposed solution is generalizable across different architectures and does not require detailed knowledge of the model’s symmetries.
3. The empirical evaluations demonstrate significant improvements in model performance and capacity, particularly in challenging scenarios like continual learning and self-supervised learning setups.

weaknesses:
1. Some of the theoretical explanations are dense and might be challenging for readers not familiar with the detailed aspects of symmetry in neural networks. Having decent knowledge of symmetries in neural networks I still struggled a bit with some sections like Section 5. Summarizing early on what the main theoretical result of each section might make it easier for reader to appreciate the contributions of the paper better. 

2. The limitations of the method are not thoroughly discussed, which could be beneficial for practitioners to know when to use this technique with caution. 

3. The formatting of the paper seems off. All the headings are pretty close to the previous paragraph hurting the readability.

questions:
1. It would be beneficial to see further exploration into the limits of the method, especially in more complex or less controlled environments than those tested. Also, maybe trying some transformer architecture based methods for SSL might make the architecture agnostic point stronger? 

2. Could the authors elaborate on any potential computational overhead or scalability issues when implementing syre in larger or more complex models?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732250768785

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary rebuttal

comment:
We thank all the reviewers for the detailed and constructive feedback. We are encouraged to hear that all reviewers find our manuscript positive, at least in some aspects. We would like to state our main contribution here, which some reviewers have misunderstood: we propose the first method that can remove almost all parameter symmetries without knowing anything about the symmetry. The method is theoretically guaranteed to work and is easy to implement (with essentially two-three lines of code).

To address the concerns of the referees, we have made the following changes and additions to the manuscript (colored in orange):

1. Removing symmetries in vision transformers in Section B6, which shows the methods can be easily scaled to the transformer architecture (Fj68, akeV)
2. Computation time/memory overhead experiment in Section B6, which shows that the proposed method has essentially no overhead in comparison to the standard training scheme when the batch size becomes large (Fj68, JSoA, akeV)
3. Empirical demonstration of the effects of Proposition 2, 3 for the science of deep learning in Section C (JSoA)
4. Additional test of the proposed method for the linear mode connectivity task, which shows competitive performance against the contemporaneous work of Lim et al. (2024), in section B7 (JSoA)



A major criticism comes from PYEZ. We believe the two points made by PYEZ are primarily due to a misunderstanding of our theoretical result and experimental settings:

1a. **The theory is too limited**. This is a misunderstanding. Our theory deals with the most general type of symmetry projections, not a restrictive subset of projections. The source of confusion is that we introduced a simplification to the definition of the symmetries. This simplification is justified in Proposition 1, which allows us to only focus on a subset of all symmetry projections without loss of generality. This is NOT a restriction of the theory.

1b. **Need to compare with weight decay**. This is also a misunderstanding. Every one of our experiments compares with weight decay (namely, the standard training method) with the same $\gamma$, which we refer to as the "vanilla" method (Figures 1, 2, 3, 5, 6, 7, 8, 9, 10, and 11). These comparisons are systematic and well-controlled in the sense that all other settings are equal. We have replaced "vanilla" with "wd" to avoid such a confusion.
</content>
</thread_object>

</thread>


Metareview:
metareview:
**Summary** This paper studies the role of neural network parameter symmetries on the phenomenon of collapse in which the model is trapped in a low-capacity state during training.  The authors provide theory explaining how symmetry can lead to reduced capacity.  That propose a model-agnostic method called syre which is a variant of weight decay which can provably break symmetry during training and empirically leads to improved performance in domains where collapse has been an issue.

**Strengths** This paper provides an interesting and insightful theory linking parameter symmetries to reduced model capacity. This theory leads to a novel method, syre, which is simple, highly general, lightweight and can avoid symmetry and collapse. The empirical evaluations show syre is effective in practice in a range of settings (including for transformers after the rebuttal). 

**Weaknesses** Reviewers initially had several criticisms, many of which were reduced by the author response.  Some reviewers considered the claims in the paper to be overstrong, but the authors reduced these claims in the revision.  Additionally, while syre is claimed to improve performance through symmetry reduction, the experiments in the paper hinged mainly on improved model performance, failing to demonstrate the exact mechanism.   The authors helped to address this by adding experiments showing syre can increase linear mode connectivity. 

**Conclusion** The reviewers and the AC have a consensus that this paper provides an insightful theory connecting parameter symmetry to reduced model capacity and an effective method to mitigate the problem.  While there are potentially other experiments which can be performed with this method, I believe accepting the paper will help encourage that follow up work.