Title: Learning (Approximately) Equivariant Networks via Constrained Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: approximate

equivariant

neural

networks

constrained

optimization
Abstract: Equivariant neural networks are designed to respect symmetries through their architecture, boosting generalization and sample efficiency when those symmetries are present in the data distribution. Real-world data, however, often departs from perfect symmetry because of noise, structural variation, measurement bias, or other symmetry-breaking effects. Strictly equivariant models may struggle to fit the data, while unconstrained models lack a principled way to leverage partial symmetries. Even when the data is fully symmetric, enforcing equivariance can hurt training by limiting the model to a restricted region of the parameter space. Guided by homotopy principles, where an optimization problem is solved by gradually transforming a simpler problem into a complex one, we introduce Adaptive Constrained Equivariance (ACE), a constrained optimization approach that starts with a flexible, non-equivariant model and gradually reduces its deviation from equivariance. This gradual tightening smooths training early on and settles the model at a data-driven equilibrium, balancing between equivariance and non-equivariance. Across multiple architectures and tasks, our method consistently improves performance metrics, sample efficiency, and robustness to input perturbations compared with strictly equivariant models and heuristic equivariance relaxations.


Peer reviews:
comments: SUMMARY:
The paper introduces Adaptive Constrained Equivariance (ACE), a constrained optimization framework that applies layer-wise homotopy for equivariance training. The key novelty is leveraging the dual problem and tighten the constraints via dual descent. The paper proposed algorithms for strictly and partially equivariant data, and provide theoretical analysis on the optimality bound. A range of experiments demonstrate the superior performance of the proposed framework, as well as analysis on training dynamics and comparison with theoretical bounds.

STRENGTHS AND WEAKNESSES:
Strength
1. The paper studies the well-motivated problem of equivariant network training, and provides novel insight with the proposed dual problem formulation.
2. The theorems provide concrete tight error bounds on equivariance performance.
3. Experimental evidence is rich and covers performance evaluation, model analysis and alignment with theoretical results.
4. The proposed framework is applicable to general network architectures.
5. Organization and presentation of the paper is clear.

Weakness
1. Assumption 1 is strong in practice, and having some discussions on its connection with real scenarios will be helpful.
2. The newly introduced step size parameters $\eta$ still needs tuning.

QUESTIONS:
1. The ACE framework introduces new step size parameters $\eta_p$, $\eta_d$. How sensitive is the method to these parameters, and what are some general rules for setting these values in practice?
2. Assumption 1 is fairly strong, for example ReLU networks are non-differentiable. So in real-data scenarios where the assumptions are violated, which aspects of the framework will be most vulnerable to failure? How robust is the framework to such violations?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-06-22

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Adaptive Constrained Equivariance (ACE), a framework for handling approximately equivariant problems. Each layer is decomposed into an equivariant block plus a non-equivariant branch scaled by $\\gamma$. Training starts with $\\gamma=1$ (fully flexible) and gradually shrinking $\\gamma$ through the constrained optimization procedure, smoothly interpolating from non-equivariant to strictly equivariant behavior. Experiments on N-body dynamics, molecular regression, shape recognition, and motion-forecasting tasks demonstrate the effectiveness of ACE.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n1. The methodology is generic and can be plugged into diverse backbone architectures to tackle approximately equivariant settings.\n2. The approach is thoroughly evaluated across multiple domains and base architectures, showcasing broad applicability.\n\nWeakness\n1. As the authors note, there are no experiments on 2-D convolutional networks. Given the wide usage of (equivariant) CNNs, such a study would greatly strengthen the paper.\n2. The paper compares against only one relaxed-equivariance baseline. Including additional methods such as [26, 27, 50] would make the empirical comparison more convincing.\n\nQUESTIONS:\n1. Would the authors consider adding a 2-D convolutional experiment that directly compares ACE with [26, 27]?\n2. The current experimental section feels crowded. Splitting it into clear subsections (for example, by data domain) would improve readability.\n2. For each task, could the authors explicitly state whether the ground-truth function is strictly or only approximately equivariant? Highlighting this distinction would help readers interpret the results.\n4. Do all experiments initialize $\\gamma$ with 1? Have the authors considered different initial $\\gamma$ values, such as starting with $\\gamma=0$?\n\nLIMITATIONS:\nAlthough the authors discuss limitations, the section is relegated to the appendix. Moving it into the main text in the final version would improve transparency and balance.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper introduces a novel technique for training approximately equivariant networks. Specifically, the networks are assumed to have an architecture consisting of a linear combination of equivariant layers with non-equivariant layers, where the weights of the latter are learned through optimization to control the degree of non-equivariance. In particular, the authors' principal contribution is a specialized training method based off of the principles of homotropy via interpretation of the objective as a generalized Lagrangian.

STRENGTHS AND WEAKNESSES:
I'm a fan of this paper. The authors contributions are well scoped, well explained, and supported by rigorous theoretical and extensive experimental results. In particular, the latter show broad applicability across a wide variety of tasks and architectures.   I think moving away from strictly equivariant models is important, as it is becoming clear that such models are ill-suited for practical applications where symmetries are often partial, messy, or otherwise corrupted.

Overall I think this paper will be a nice contribution to the conference and I am happy to recommend clear acceptance. That said, I have some minor questions/concerns below.

QUESTIONS:
The key question I have is regarding the implementation. It seems like the application of Algorithm 2) would require a modification of a standard optmizer (such as Adam/AdamW).  Perhaps I missed it, but it appears that this is not discussed in the paper. 

Is it necessary to modify the optimizer to handle the proposed training method? If not that is a big benefit and should be stated. If not, then a detailed explanation of a general recipe should be stated in the main paper or appendix, in addition to the release of the authors code.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIt is well known that equivariant models cause complex loss functions even if the data obey symmetry and lower the expressiveness of the model. The authors propose Adaptive Constrained Equivariance: an optimization framework that starts from a non-equivariant model and introduces gradually equivariance via homotopy enabling thus a smooth transition from a non-equivariant to an equivariant model.\n\nSTRENGTHS AND WEAKNESSES:\nS1: The first ever optimization that automatically transitions from a free to an equivariant model.\n\nS2: Theoretical guarantees on the equivariance approximation.\n\nS3: It avoids manual tuning of losses and optimization schedules\n\nS4: The authors cleverly exploited the fact that dual methods in constrained optimization are instances of homotopy continuation. This is explained well by starting with the dual of the pure equivariant problem, namely the empirical Lagrangian, which approximates the solution of the equivariant problem. Algorithms 1 and 2 juxtaposed illuminates very well the difference.\n\nS5: Stopping the iteration over the non-equiv contribution $\\gamma$ is formally shown to produce a bounded equivariance error if both equivariant and non-equivariant models are Lipschitz continuous.\n\nS6: The authors moreover show that when data are partially summetric the non-equivariant model approximately satisfies the equivariance constraint with a proven bound. \n\nS7: The trade-off between performance and equivariance can be resolved with a resilient constrain loss based on slacks. \n\nS8: It is awesome that the authors formalize three questions/hypotheses for the experiments.\n\nQUESTIONS:\nQ1: L. 214 Not clear what it means by bounded translation. \n\nQ2: It is not clear what symmetry was assumed in the MoCap dataset. What is the homogeneous space and which is the transitive action? Is it a global SO(3)?\n\nQ3: While the gamma and loss plots give some insight, I think a very simple example with two weights and a very easy symmetry like SO(2) could give a very clear insight in the studied behaviors.\n\nLIMITATIONS:\nnone identified.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel training framework that overcomes the limitations of training strictly equivariant models by treating equivariance as a soft constraint and utilizing techniques from the constraint optimization literature. Specifically, the proposed Adaptive Constraint Equivariant (ACE) framework augments the equivariant model with per-layer non-equivariant unconstrained layers for which the output is modulated by parameters $\\gamma$. The problem of training an equivariant network is then expressed as a constraint optimization problem, where the equivariant constraint is reduced to the equality constraint of $\\gamma=0$. The authors then propose to optimize the original problem by optimizing the dual problem via gradient descent on the model parameters and gradient ascent on the dual variables. Additionally, the authors provide supporting theoretical analysis of the approximation error that occurs when the network returns to the equivariant parametrization and the end of training, by setting $\\gamma=0$, and describe a mechanism that can allow users to infer possible symmetry misspecification by looking at the convergence of the $\\gamma$ variable. Finally, the experimental section provides empirical evidence demonstrating the benefits of the proposed ACE framework for a wide range of equivariant tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The authors propose a task-agnostic framework that, in contrast with previous works, doesn't rely on a user-specified learning schedule and allows for a data-dependent modulation of the equivariant constraint during training.\n- The framework can be applied to both equivarariant and approximate equivariant tasks.  Section 4.2 provides a nice heuristic for determining what symmetries are part of the tasks and whether a fully equivariant or approximately equivariant approach is more beneficial.\n- The formulation of the homotopic architectures requires minimal assumptions about the type of model to which the ACE framework can be applied.\n\nWeaknesses:\n- While the authors claim that their framework doesn't require additional user tuning, the optimization of the dual problem introduces additional hyperparameters (e.g. the learning rate for the dual variable $\\lambda$). There is no study of the sensitivity of the optimization to these additional parameters.\n- There is a gap between the accuracy reported for VN-DGCNN in Figure 7 and the accuracy reported in the original paper for ModelNet40. Additionally, in Figure 7, there is a small typo in the name of the model used.\n- In the homotopic architecture, the non-equivariant component is not constrained, and although it is a bounded operator by B, B is free to become arbitrarily large during training to counterbalance the reduction of $\\gamma$. This process can increase the approximation error shown in Theorem 4.1, even from small $\\gamma$ values.\n- Minor: There is an inconsistency in the naming of the different methods between Figure 2 and Figure 7. In Figure 2, the normal SEGNN (which is an equivariant model) is referred to as \"No Constraints\" model, while in Figure 7, the equivariant VN-DGCNN is referred to as \"Equivariant\" model.\n\nQUESTIONS:\n- What is the sensitivity of the optimization of the dual problem to the choice of learning rate for both the parameters of the model and the dual variables?\n- Is it possible that the non-equivariant component of the homotopic architecture increases the scale of its outputs during training as a way to counteract the reductions of the $\\gamma$ variable?\n- What is the reason for the gap between the reported performance of VN-DGCNN in this paper and the performance reported in the original Vector Neurons paper?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}