Learning (Approximately) Equivariant Networks via Constrained Optimization
Authors: Andrei Manolache
Luiz F. O. Chamon
Mathias Niepert
Venue: NeurIPS 2025 oral ()
Keywords: approximate
equivariant
neural
networks
constrained
optimization
Primary area: deep_learning
TL;DR: We introduce Adaptive Constrained Equivariance, a homotopy-inspired constrained optimization apprach for training equivariant neural networks.

Abstract: Equivariant neural networks are designed to respect symmetries through their architecture, boosting generalization and sample efficiency when those symmetries are present in the data distribution. Real-world data, however, often departs from perfect symmetry because of noise, structural variation, measurement bias, or other symmetry-breaking effects. Strictly equivariant models may struggle to fit the data, while unconstrained models lack a principled way to leverage partial symmetries. Even when the data is fully symmetric, enforcing equivariance can hurt training by limiting the model to a restricted region of the parameter space. Guided by homotopy principles, where an optimization problem is solved by gradually transforming a simpler problem into a complex one, we introduce Adaptive Constrained Equivariance (ACE), a constrained optimization approach that starts with a flexible, non-equivariant model and gradually reduces its deviation from equivariance. This gradual tightening smooths training early on and settles the model at a data-driven equilibrium, balancing between equivariance and non-equivariance. Across multiple architectures and tasks, our method consistently improves performance metrics, sample efficiency, and robustness to input perturbations compared with strictly equivariant models and heuristic equivariance relaxations.

Learning (Approximately) Equivariant Networks via Constrained Optimization
Andrei Manolache Luiz F.O. Chamon' Mathias Niepert
CCmppter Science Department, University of Stuttgart, Germany International Max Planck Research School for Intelligent Systems, Germany Department of Applied Mathematics, École Polytechnique de Paris, France "Bitdefender, Romania
(andrei manolache, mathias niipert) 0ki uni-stuttgart.dd luiz. chamon@@olyytechniqqus edu
Abstract
Equivariant neural networks are designed to respect symmetries through their ar- chitecture, boosting generalization and sample efficiency when those symmetries are present in the data distribution. Real-world data, however, often departs from perfect symmetry because of noise, structural variation, measurement bias, other symmetry- symmetry-breaking effects. Strietly equivariant models may struggle to the data, while unconstrained models lack a principled way to leverage partial symmetries. Even when the data is fully symmetric, enforcing equivariance can hurt training by limiting the model restricted region of the parameter space. Guided by homotopy principles, where an optimization problem is solved by gradually trans- forming simpler problem into a complex one, we introduce Adaptive Constrained Equivariance (ACE), a constrained optimization approach that starts with flexible, non-equivariant model and gradually reduces its deviation from equivariance. This gradual tightening smooths training early on and settles the model at a data-driven equilibrium, balancing between equivariance and non-equivariance. Across multi- ple architectures and tasks, our method consistently improves performance metrics, sample efficiency, and robustness input perturbations compared with strictly equivariant models and heuristic equivariance relaxations.
Introduction
leverage known symmetries improve sample efficiency and generalization, and are widely used in computer vision 1357, graph learning [А]]) and 3D structure modeling [2.17.. Despite their advantages, training these networks can be challenging. Indeed, their inherent symmetries give rise to complex loss landscapes (18) even when the model matches the underlying data symmetry [8
Equivariant neural networks (NNs) 1+4
Additionally, real-world datasets frequently include noise, measurement bias, and other symmetry- breaking effects such as dynamical phase transitions or polar fluids 22 24 that undermine the assumptions of strictly equivariant models (25-27) These factors complicate the optimization of equivariant NNs, requiring careful hyperparameter tuning and often resulting in suboptimal performance.
well-established strategy for approaching difficult optimization problems to use homotopy (or continuation), i.e., first solve simple, surrogate problem and then track its solutions through sequence of progressively harder problems until reaching the original one (28, 29). Continuation has been used for compressive sensing (30)32) and other non-convex optimization problems 35) where it is closely related to simulated annealing strategies [36 These approaches suggest that introducing equivariance gradually can facilitate training by guiding the model through a series
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

MW
(x)
and show fa (m)
(x)
Fmi (x)
Su (mr)
(1)
fon (x)
71
MV
72
MV
7/2 vm
73 WW
Y/3 mm
Figure 1: The proposed ACE homotopic optimization scheme at glance. Left: The red trajectory illustrates the training of relaxed, non-equivariant model that gradually becomes equivariant as the layer-wise coefficients " decay. The blue trajectory illustrates a strictly equivariant network feq trained from the same initialization. Right: the coefficient 7 allows the training problem transition between easier (non-equivariant) and harder (equivariant) problems, facilitating convergence.
of less stringent intermediate optimization problems. However, implementations of this strategy based on modifying the training objective [8] or the underlying model [41] require domain- specifcc knowledge to design the intermediate stages and craft loss penalties, limiting practical usability.
This work, therefore, addresses the following question: is there automated, general-purpose purpose strat- egy to navigate these intermediate problems that precludes manual luning and that works effectively for various modely and tasks? To this end, leverage the connection between homotopy approaches and dual methods in optimization, which both provide rigorous frameworks for solving constrained problems by addressing a sequence of relaxed problems with progressively stricter penalties. By cast- ing the training process as a constrained optimization problem, we introduce Adaptive Constrained Equivariance (ACE), principled framework for automatically relaxing/tightening equivariance in NNs. Our method balances downstream performance (e.g., accuracy) with equivariance, relaxing equivariance at initial stages while gradually transitioning strictly equivariant model, as illus- trated in Fig. [ This transition occurs in an automated, data-driven fashion, eliminating much of the aall-an-emma associated with manual tuning.
We summarize our contributions as follows:
1. A general equivariant training framework. We introduce ACE, principled constrained optimization method train equivariant models by automatically transitioning in data- driven fashion from flexible, non-equivariant model to one that respects symmetries without manually tuned penalties, weights, or schedules (Sec.
2. Accommodating partial equivariance. Our data-driven method can identify symmetry- breaking effects in the data and automatically relax equivariance to mitigate their impact downstream performance while exploiting the remaining partial equivariance (Sec. 4.2)
3. Theoretical guarantees and empirical alignment. We provide explicit bounds on the approximation error and equivariance level of the resulting model (Thm. how they are reflected in practice.
4. Comprehensive empirical evaluation. We conduct systematic study across four represen- tative domains to examine the impact of our algorithm on convergence, sample efficiency, and robustness input degradation compared to strictly equivariant and relaxed alternatives.
Related Work
Equivariant NNs encode group symmetries directly in their architecture. In computer vision, for instance, equivariant convolutional NNs accomplish this by tying filter weights for different rota-
tions and reflections 04 Equivariant NNs have also been extended 3D graphs and molecu- lar systems, is the case of SchNet, that operates directly atomic coordinates while remain-

---

ing E(3)-invariant [12 and Tensor Field Networks or SE(3) Transformers, that extend this to full roto-translation equivariance using spherical harmonics and attention, respectively [3 42
Scalable, parameter-effiient architectures that remain equivariant include EGNNS
43 SEG-
NNs 44 and methods such Vector Neurons 455 which lift scalars 3ID vectors, achieving exact (3) cquivariance in common layers with minimal extra parameters. Moreover, recent works model
3D trajectories rather than static snapshots, such as EGNO, which uses SE(3) equivariant layers 46
combined with Fourier temporal convolutions to predict spatial trajectories
While exact equivariance can improve sample efficiency [47 1, it may render the optimization harder, particularly when the data violates symmetry (even if partially) due to noise or other symmetry- breaking phenomena [22 24 This has motivated the use of models with approximate equivariance based on architectures that soften weight-sharing. such as Residual Pathway Priors [50] that adds non-equivariant branch alongside the equívariant one, works such Wang et al. (26.
relaxed
where weighted filter banks are used to interpolate between equivariant kernels. Recent works have formalized the trade-off's that appear due these symmetry violations, establishing generalization
bounds that specify when and how much equivariance should be relaxed [19]
Beyond architectural modifications, another line of work relaxes symmetry during training. REMUL [8] adds a penalty for equivariance violations to the training objective, adapting the penalty weights throughout training to balance accuracy and equivariance. This procedure does not guarantee that the final solution equivariant or even the degree of symmetry achieved. An alternative approach modifies the underlying architecture by perturbing each equivariant layer and gradually reducing the perturbation during training according to a user-defined schedule [41]. This guided relaxation facilitates training and guarantees that the final solution is equivariant, but is sensitive to the schedule. Additional equivariance penalties based on Lie derivative are used to mitigate this issue, increasing the number of hyperparameters and restricting the generality of the method.
The method proposed this paper addresses many the challenges in these exact and relaxed equivariant NN training approaches, namely, (i) the complex loss landscape induced by equivariant NNs; (ii) the manual tuning of penalties, weights, and schedules; (iii) the need to account for partial symmetries in the data.
3 Training Equivariant Models
Equivariant NNs incorporate symmetries inherent to the data by ensuring that specific transformations of the input lead to predictable transformations of the output. Formally, let fo : X - Y be a NN parameterized by 0 E RP, the network weights. A NN is said to be equivariant to a group G with respect to actions px on X and py on Y if
fo (px((g))= pr (9) for all X and g E G.
(1)
Equivariant NNs improve sample efficiency by encoding transformations from a symmetry group G 44]. Strictly enforcing equivariance, however, limits the expressiveness of the model and create intricate loss landscapes, which can slow down optimization and hinder their training [19 [8]. particularly in tasks involving partial or imperfect symmetries (4)
To temper these trade-offs, recent work relaxes symmetry during training by modifying either the Y be a homotopic parameters representing the model non-equne non-equivariance. The model is therefore equivariant for ry = 0, i.e., fe.o satisfies (I). but can behave arbitrarily if hyl > 0 for any Consider training this NN by solving
optimization objective or the model itself 44 18
26 19
Explicitly, let fo.y
NN now parameterized by both 0, the network weights, and > RR,,
N
minimize
Σ n=1
(o( + Blog (Fe.t.
(2)
N
where (In. Vn) are samples from data distribution D. lu denotes the top-line objective function (typ- ically, mean-squared error or cross-entropy), fag denotes an equivariance metric, and o, > 0 are hyperparameters controlling their contributions to the training loss. Hence, traditional equivariant NN training amounts to using a = and B = = 0 in (2) In [8], a non-equivariant model (> = 1) is trained using (f) Σ., E... [lIf J..) PY (f with gn sampled uniformly
3

---

from G. The values of a. 3 are adapted to balance the contributions of each loss, i.e., to balance accuracy and equivariance. In contrast, [4] uses an equivariance metric Lay based on Lie deriva- tives [51 52, fix a, 8, and manually decrease y throughout training until it vanishes, i.e., until the model is equivariant.
While effective in particular instances, these approaches are based on sensitive hyperparameters, some of which are application-dependarr
This is particularly critical in [18), where both performance and equivariance depend on the values of ax 3. While [1 guarantees equivariance by driving to zero, its performance is substantially impacted by the rate at which it vanishes. It therefore continues to use fey even though it is not needed to ensure equivariance. What is more, it is once again susceptible to soametty--reakkngg effects in the data since strictly imposes equivariance. In the sequel, we address these challenges by doing away with hand- hand-designed penalties (3 = 0) and manually-tune tunned schedules for y. Before proceeding, however, we briefly comment on the implementation of so.r-
Homotopic architectures. Throughout this paper, we consider an architecture similar to [4 built by composing equivariant layers linearly perturbed by -quivvriar components, e.g., a linear layer or a small multilayer perceptron (MLP). Formally, let
San So o... for with fo. - 1 p - 1 nog.
L.
(3)
where 7 = :.).: f²: Zi-1 + Z; (and thus similarly for fou and freg.),, with Zo X and Zr ;; and poyil equivariant with respect to the actions Pi of group G on Zi. i.e., ((())) for all g E G. Note that Pa = Px and PL = -PY. We generally take Z, Rd and the group action Pr to be map from G' to the general linear group GL3, (i.c.. the group of invertible & x K2 matrices). Note that although we use the same set of parameters for all layers, they need not share weights and can use disjoint subsets of 0. Immediately, the NN is equivariant for 7 = 0, since is a composition of equivariant functions foy Otherwise, it is allowed to deviate from equivariance by an amount controlled by the magnitude of y. It is worth noting, however, that the methods we put forward in the sequel do not rely on this architecture and apply to any model fory that is differentiable with respect to & and y and such that fn. is equivariant.
for (p: (g)2)
4 Homotopic Equivariant Training
In this section, we address the challenges of equivariant training and previous relaxation methods, namely (i) the complex loss landscape induced by equivariant NNs (7 = 0); (ii) the manual tuning of penalties, weights, and/or homotopy schedules; (iii) the detection and mitigation of partial symmetries in the data. We use the insight that dual methods in constrained optimization are closely related to homotopy or simulated annealing in the sense that they rely on a sequence of progressively more penalized problems. This suggests that (i)- can be addressed in principled way by formulating equivariant NN training as constrained optimization problem (Sec. We can then analyze the sensitivity of dual variables to detect the presence of symmetry violations in the data and relax the equivariance requirements on the final model [Sec. 4.2 (iii). We also characterize the approximation error and equivariance violations of the homotopic architecture (3) (Thm.
4.1 Strictly equivariant data: Equality constraints
We begin by considering the task of training a strictly equivariant NN, i.e., solving 2 with = 1
and / = y = Note that this task is equivalent to solving
minimize 0,7
B(Zw)~ lo (t))),
(PI)
subject to 7/ 00 for i 1,...,, L.
Indeed, only equivariant models fe.o are feasible for (PI) so that its solution must be equivariant. While (PI) may appear to be a distinction without a difference, its advantages become clear when we consider solving using duality. Specifically, when we consider its empirical dual problem
N
I.
maximize min L(0, 7, A) XERL
A
+
Ar
(DI)
0,7
N

---

Algorithm 1 Strictly equivariant data > 0, 10)
Algorithm Partially equivariant data
1: Inputs: Up- Ndd
1()
Inputs: Opp Id > 0, (0) =1, 1()
N
2: J N n=l
Eo (occ Hr)
2: _
Eo (fcco (xx))
N n=1
3: ((++1) = (())
7p Vo (1)
((++1) g(()
np Ve-
4: 7/ 7
((+1) (1)
1/p V J0 (0) + (00)
4: 7 7
((+1) (t)
1hp V 70 + (()
(1)
5: ((+1) (t)
+ np (p (t) - -1(0)
u
5: x((-1) 11th + nd7
(8)
1 ((-1) [ + na(l) 10- ]
where (In, are i.i.d. samples from D. À E I collects the dual variables 153 54 A and i
is called the empirical Lagrangian 55 56 In general, solutions of (PI) and (DI) not related
due to (i) non-convexity, which hinders strong duality [57). and (ii) the empirical approximation of the expectation in (PI) by samples. Yet, for rich enough parametrizations fe.y. constrained learning
theory shows that solutions of (DI) do generalize to those of (PI) (53/55]. In other words, (Di)
approximates the solution of the equivariant NN training problem PI without constraining the model to be equivariant.
More precisely, DD gives rise gradient descent (on 7) and ascent (on X) algorithm (Alg. D that acts as an annealing mechanism adapted to the downstream task, taking into account the effect of
enforcing equivariance on the objective function. Indeed, by taking 7 (0) 1, Alg. starts by training a flexible non-equivariant model (step 3.. The gradient step on the homotopy parameters %/2 (step then simultaneously seeks reduce the objective function lu and bring "/2 closer zero, targeting the constraint in (PI). The strength of that bias toward zero (namely, X₂) depends on the accumulated constraint violations (Stepp): the longer Y is strietly positive (negative), the larger the magnitude of Ar. Hence, the "r can increase to expand the flexibility of the model or decrease to exploit the symmetries in the data. The initial non-equivariant NN is therefore turned into equivariant one by explicitly taking into account the effects this has on the downstream performance, instead of relying on hand-tuned, ad hoc penalties or schedules as in 18 41)
Naturally, the iterates 7/ do not vanish in any finite number of iterations. In fact, the behavior of saddle-poing algorithms such as Alg.
A aree intricate even in convex settings [58 [9] and its iterates may, even for small learning rates Tpp Nd. (i) asymptotically approach zero or (ii) display oscillations that do not subside. In case (i), if we stop Alg. at iteration T and deploy its solution with 0 (i.e.,
fecr),)) .)., we introduce an error that depends on the magnitude of 7/ (T). The following theorem bounds this approximation error for the architecture in (3) under mild assumptions on its components. In particular, note that Ass. [ holds, e.g., for MIPs with ReI.U activations and weight matrices of bounded spectral norms 660
Assumption 1. The architecture in 3 is such that, for all 0 and i, 18 and pooqss are M-Lipschitz continuous and faeq. is bounded operator, i.e., |1 (z))l < B||x||| for all x e X.
Theorem 4.1. Consider Jo.y as in 3) satisfying Assumption 7 1f7 max; Inl, then
[[) - fo,oo <
Σ (1 + 3) 3BM
||||| for all x E X.
proof of this as well as a more refined bound is available in the Appendix that as long as the (,,"²) are small enough, the error incurred from setting them to zero is negligible. This is not the case, however, when the 7, oscillate without subsiding [case (ii)). As we argue next, this effecc can be used to identify and correct for symmetry deviations in the data.
A.6 Theorem 4.1 shows
5

---

4.2 Partially equivariant data: Resilient constraints
In some practical scenarios, the data used for training may only partially satisfy the equivariance relation (T) due to the presence of noise, measurement bias, or other symmetry- brreakktbbbbkkingggg effects [20]- 277. In such cases, the equivariance constraints in (PI) become too stringent and the gradient of the objective with respect to 7 in step 4 of Alg. T does not vanish as 7/ approaches zero Alg. [ will then oscillate as steps| and 5 push 7i alternatingly closer and further from zero [case (ii) or "/ will settle, but X will not vanish. These observations can then be used as evidences that the data does not fully adhere to imposed symmetries. This is not necessarily a sign that equivariance should be altogether abandoned, but that we can improve performance by relaxing it. To this end, we rely on the fact that y provides a measure of non-equivariance, which we formalize next.
Theorem 4.2. Consider far as in (3) satisfying Assumption 7 and suppose that the group actions are bounded operators, i.e., for all it holds that (|p:(g))11 S B || for all E G and 2 E Zi. I - max; had and C a max B, 1), then
(0)
s 27 (M + C-) LB for all X and g E G.
Once again, the proof of a more refined bound can be found in Appendix A.6| Also note that the bounded assumption the group action is mild and that in many cases of interest the group action is in fact isometric, i.e. [g)[1] = ||=|| (e.g., rotation, translation). Theorem 4.2 shows that by adjusting the degree to which the equivariance constraint in (PI) is relaxed, we can control the extent to which the final solution remains equivariant. While we could replace the equality in (PI) by Ias S Uy for u; 2 O, choosing a suitable set of {u,} } is not trivial, since they must capture how much partial equivariance we expect or can tolerate at each layer. Setting the u, too strictly degrades downstream performance (objective function Eo), whereas setting them loosely interferes with the ability of the model to exploit whatever partial symmetry is in the data. Striking this balance manually impractical, especially as the number of layers (and consequently 7/) increases.
Instead, we use the outcomes of Alg. T to determine which constraints in (PI) to relax and by how much. Indeed, notice that we only need relax those "/ that do not vanish, seen as they are the ones
too strict for the data. We can then leverage the fact that stricter constraints lead to larger A [53]5]] (57) and relax each layer proportionally to their corresponding dual variable. In doing so, reduce the number of 7/2 * O as well as their magnitude. thus improving downstream performance (as measured by Eo) while preserving equivariance as much as possible (Theorem 4.2))
This trade-off rationale can be formalized and achieved automatically without repeatedly solving Alg. by using resilient constrained learning [53
54 61. Concretely, we replace (PI) by
minimize 0,7.0
B(x to (fe. v) + 2 |lu||
(PII)
subject to hyd S us. for = L.
where u E Rt collects the slacks and p > 0 is proportionality constant (we use 1 through- out). Notice that the slacks are not hyperparameters, but optimization variables that explicitly trade off between downstream performance, that improves by relaxing the constraints (i.c., increasing u,). and constraint violation (i.e., the level of equivariance). In fact, it can be shown that a solution of (PI)) satisfies in = X /p. where > is the Lagrange multiplier of the corresponding constraint [53] Prop. 3]. Using duality, (PII) lead to Alg. 2 where [=]+ mmx(2; 2, 0) denotes a projection onto the non-negative orthant. Notice that it follows the same steps andd of Alg. However, it now contains
an additional gradient descent update for the slacks (step 5) and the dual variable updates (step 6) now has projection due to the constraints in (PII) being inequalities [57]. During training, Alg 2 therefore automatically balances the level of equivariance imposed on the model (u,) and the relative difficulty of imposing this equivariance (A.) opposed to improving downstream performance (Jo).
5 Experimental Setup and Results
In this section, we aim to answer the following three research questions:
RQL. Final Performance: How does gradually imposing equivariance via ACE affect downstream task accuracy and sample efficiency across diverse domains and models?

---

Val MSE Mcan + Std Dev (3000 samples)
Test MSE (Mean + Std Dev) vs. Samples
9 10
Equality Constraint ()
0.006
Equality Constraint (fa)
Val MSE
8 10
x
Bas: Model
Test MSG
0.005
Base Model
6x 10
0.004
10
Val MSMM
MSE
0.003
x 10
500
1000
1500
2000
1000
3000
5000
7000
9000
Epoch
Number of Samples
Figure 2: SEGNN trained with ACE equality constraints compared with the normal SEGNN on the N-Body dataset. Left: Validation MSE over 2000 epochs. Right: Test MSE versus training set size.
RQ2. Training Dynamics and Robustness: How do ACE constrained equivariance strategies, namely equality constrained optimization and partial equivariance enforced via resilient inequality constraints, affect convergence under input degradation, guide predictive per- formance when relaxing strict equivariance, and reveal where flexibility is most beneficial through their learned parameters?
RQ3. Theory vs. Empirics: To what extent do the theoretical equivariance-error bounds (Theo- predict the actual deviations observed during training?
4.2)
5.1 Final Performance and Sample Efficiency
N-Body Simulations. To address RQ1, we begin with the N-Body simulations dataset (63] In Fig- (left,, we compare the validation MSE (meandssd)) across 2000 epochs for the "vanilla" SEGNN and an equivariant projection f obtained by training with ACE. Remarkably, f trained with ACE converges more rapidly and attains a lower final MSE than the vanilla baseline, demonstrating both accelerated training dynamics and greater stability. Figure 2 (right) shows test MSE with respect to the training set size. Here, our f trained on only 5000 samples matches (and slightly exceeds) the vanilla SEGNN's performance at 9000 samples, corresponding to = 44% reduction in required data. Table 2 (left) presents test MSE results (x10- -3) for various SEGNN variants and baselines. Our SEGNN N trained with ACE, feq, outperforms all other baselines, including the heuristic- based relaxed equivariance scheme of Pertigkiozoglou et [1] and, after projection, matches the performance of non-equivariant SEGNN that retains the same architecture foa + fneg but, without using ACE, relies on explicit 5O(3) augmentations to learn equivariance.
Molecular Property Regression. We evaluate on the QM9 dataset 64 65
using the invariant SchNet [12] and our ACE variant. Due to its strict invariance, SchNet may severely hinder conver- gence without any relaxation; indeed, as shown Table 2 (right), incorporating ACE during training reduces the mean absolute error (MAE) across all quantum-chemical targets. To assess the impact of ACE on a more flexible architecture, we compare standard SEGNN 44] against its ACE counterpart on QM9. As shown in Table (right), the ACE SEGNN lowers the MAE on most targets; however, the overall performance remains largely comparable. Collectively, these findings show that SEGNN's E(3i--quiva aligns well with the QM9 dataset, facilitating optimization. However, ACE can still provide an additional boost in performance for certain molecular properties.
3D Shape Classification. On the ModelNet40 classification benchmark [66], apply ACE the VN-DGCNN architecture [5] Figure)) (left) shows that our ACE equivariant model f achieves gains of +2.78 and 1.77 percentage points for the class and instance accuracy metrics over normal training. The partially equivariant ACE variant fey + faeq achieves comparable improvements. Both models obtained by checkpointing the same training runs.
5.2 Training Dynamics and Robustness
Noisy 3D Shape Classification. To address RQ2, we first evaluate convergence under input degradation on MedelNet40 (66) by randomly dropping 0-85% of the points the 3D point clouds of every shape. As shown in Figure 7 (right), the strictly equivariant baseline fails to converge under this
7

---

Iquivariant
Test MSE (x10-2) on the CMU MoCap dataset for Subject #9 (Run) and Subject #35 (Walk). Results are shown for the standard equivariant mod- els, EGNO models trained with an equality constraint (ACEf) and an EGNO model trained with an resilient inequality constraint (ACEjj. The partially equivari- ant EGNO ( fueg) outperforms both the standard and equality- connstii-cotttaineed projection f models, with the adlativv-rrssileen variant achieving the lowest
Table 1:
Equality Constraint
Resilience Constraint
error on both subjects.
Model MPNN
MSEJ (Run)
MSEJ (Walk)
66.4=22
36.1+15
RF 62
521.3=23
188.0+1.
Qualitative comparison on the CMU MoCap dataset (Subject #9, Run). Top row: Standard equívariant EGNO. Middle
Figure 3:
THN [3
56.6=1.
32.0#118
SE TT. 42
61.2=23
31.5+2.1
EGNN
43
50,9
28.7:116
EGNO (report.)
46
33.9
8.1
row: Partiall equivariant EGNO (f ((ee)) trained with an ACE equality constraint. Bot- tom row: Partially equivariant EGNO (f + faee) trained with ACE resilient inequality constraint, yielding significant improvements over both.
EGNO (reprod.) 46
35.3=32
8.5+
EGNO ACEI
35.3
7.9
EGNO ACE!
32.6=1.6
7.5
ACE! EGNO you foom
23.8=115
7.4+0.2
Resilience Constraint Validation Equivariance Error
Un
U1
Equivariance Eror
Equivarianes Error
Value
Value
30-
Epoch
Epoch
U10
4/11
Vilue
Value
100
200
300
400
500
600
200
400
600
200
400
600
Epoch
Epoch
Epoch
Figure 4: Validation equivariance error dur- ing training of partially equivariant EGNO model with an ACE resilient inequality con- straint on the CMU MoCap dataset (Subject #9, Run). The equivariance error decreases and approaches values near zero.
Figure 5: Adaptive constraint values u on the first
two and last two layers of EGNO model trained with ACE resilient inequality constraints on the CMU MoCap dataset (Subject #9, Run). Early layers allow large slacks that decrease over training, while later layers stay more tightly constrained.
perturbation, whereas our equality-constraiieed projection feg converges quickly to substantially higher test accuracy.
Relaxed Equivariance for Motion Capture. Next, we evaluate predictive performance on the CMU Motion Capture (MoCap) "Run" and "Walk" datasets [69]. where we examine the effect of training with both the equality constraint and the incquality constraint of resilient constrained Icarning. As shown in Table the equivariant projection for performs similar to the vanilla EGNO. However, the partially equivariant for + fueq outperforms both models, indicating that strictly equivariant models might not align well with the dataset. To examine whether better predictive performance can be obtained with partially equivariant model, we perform experiments with resilient inequality constraints and observe that this training strategy further reduces test MSE on both Run and Walk compared to the purely equivariant model and the ones trained with equality constraints. These results confirm that resilience parameters can adaptively allocate non-equivariant capacity where is most needed, improving overall performance.
Modulation Coefficients Dynamics. To better understand the effects of ACE, we inspect how the constrained parameters evolve during training. On ModelNer40, the modulation coefficients The in the equality-constraneed VN-DGCNN decay rapidly and reach zero within the first third

---

Yo
Ys
Yo
Y2
Voluoo
Value
Value
Valle
www.mm
mwmmmm
Epoch
Epoch
Epuch
Epoch
Y11
V13
V4
Vs
Value
Value
Vmmm
Value
Value
w
100
200
0
100
200
1000
2000
1000
2000
Epoch
Epoch
Epech
Epoch
Figure 6: Evolution of y in ACE equality constrained networks. Left: VN-DGCNN on ModelNet40;
layers 0, 5, 11, 13 decay to zero, with deeper layers briefly oscillating, evidence that relaxing equivariance in later layers improves the final solution. Right: EGNO on CMU MoCap (Run); layers 0, 2, 4, 5 hover around zero without converging, showing that strict equivariance obtained from training with equality constraints is too restrictive.
Noisy ModelNet40 Test Accuracy
DGCNN Variant
Class Acc. f
Inst. Acc. f
0.75
Accuncy
DGCNN-VE VN
65
0.7707
0.8223
0.50
DGCNN- VN FALL rang
ACET
0.791910.0156
0.0395
0.25
Base Model
ACET DGCNN-VN
0.7085
0.8400
Equality Constraint (PS)
0.00
100
150
Epech
Figure 7: Accuracy and robustness of normal and equality VN-DGCNNs. Left: On the clean ModelNet40, partially and purely equivariant constrained models (ACE+) outperform the baseline. Right: On noisy ModelNet40, training without ACE is unstable, while the equivariant model obtained with ACE (fe9) yields stable, high-accuracy solution.
of training (Figure 6 left). In the early layers. y rapidly decays to zero; in the deeper layers, oscillates around zero, indicating that the optimizer initially explores non-equivariant directions before settling on strictly equivariant solution that aligns with the data symmetries. contrast, the same coefficients in EGNO oscillate around O throughout training on the MoCap dataset, suggesting that strict equivariance may overly restrictive for this task (Figure 6 right). The learned slack values ug in the resilient model further support this view: as shown in Figure 5 early layers acquire larger slack, allowing ficxibility where local structure dominates, while later layers remain nearly equivariant. As training progresses, the constraints tighten for all layers, increasingly penalizing non non-equivariant solutions. We provide plots for all the layers in Appendix A.8 These dynamics demonstrate the ability of the proposed constrained learning framework to guide the model toward suitable, approximately equivariant behavior, stabilizing training under degradation, and improving performance over fully equivariant models.
5.3 TTheory-Empiric Alignment
To address RQ3, we analyze how equivariance aligns with observed model be- havior in the ACE resilient setting.
We compute the equivariance deviation as Ey-so(3) [lley (x) fe.- (px (g)x) [I] (cf. Theorem 4.2). where the expectation is approximated using five random samples g E SO(3). As shown in Figure 4 the deviation from cquivariance measured for the resilient model steadily decreases and approaches values near zero by the end of training. This suggests that the equivariance constraint is effectively enforced in practice, and that partial equivariance can be preserved in a stable and interpretable way throughout training.
Limitations and Further Work
While our ACE framework is broadly applicable to a range of architectures, does incur some overhead: during training we introduce extra eooi-vauivariant) MLP layers, which modestly slow down training (see Appendix Table Addressing this, one avenue for future research is to "break" equivariance without adding any auxiliary parameters, e.g., by integrating constraint enforcement directly into existing layers or by designing specialized update rules that can break equivariance while preserving runtime efficiency.

---

Table Error metrics for equality-constrained projections reach the lowest test MSE (x10-3). ACEi represents models trained with equality constraints, * are trained without ACE, but with SO(3) augmentations, Right: QM9, constraining SchNet and SEGNN reduces or matches MAE on most quantum-chemica targets.
constrained models. Left: N-body, equality constriee SEGNN
Method
MSE
SchNet 12 (MAE .)
SEGNN 44
(MAF
Linear
45
81.9
Target
Base
ACE (/*9)
Base
ACE ( ((89)
SE(3)-Tr.
42
24.4
0.051 00511:00000 +000
0.0393:10.000
0.0371 000000
0.0362:00.007
RF 62
10.4
ClofNet
70
6.5
a
Ebomno
0.0500
0.1236++00037
0,0443:10.006
0.0952.+0.004
0.0250=0.00m
0.0991 0.00033
0.0936=000126
0.0244=00000
EGNN
43
7.1
0.0421 +:00005
0.0360
0.0244
0.0257 #000006
EGNO
46
5.4
As
0.0758++0.000
0.0686::0.011
0.0450
0.0505
SEGNN 44
5.6.1025
(R*)
1.4486 1225
0175355:0.4433
1.2022
1.1681 10.19
SEGNN
Rel. 41
4.9.0..18
ZPVE
0.00299:0.000
0.0018 0.00000
0.0029
0,0025 0.0001
SEGNN
5.7:021
Up
9.6429#3.2675
9.400391..2247
2.6163=004022
2.5597
0007760
SEGNN
3.8::.066
U
8.3412
3,3540 1.05000
2.5645
2.4796
1:014400
SEGNN
ACET
yog+ food
3.8
H
9.5835 1029
3,4403-10.4440
2.7577
2.4569
005566
8.5405
3.6473
2.7505=006970
2.5954
1.0200
SEGNN
ACE! for
3.8:1..17
Cr.
0.007
0.0376:10.0066
0.0389
0.0403
5000000
Moreover, our experiments have focused on symmetry groups acting on point clouds, molecules, and motion trajectories. A natural extension is to apply our approach to other symmetry groups and data modalities, such as 2D image convolutions or 2D graph neural networks. As an initial step in this direction, our Ca-symmetric 2D convolution toy experiment (see Appendix (A.4) shows that ACE adapts appropriately, driving 7 toward zero when the target matches the imposed symmetry, and increases " when correcting for symmetry mis-specification, suggesting feasibility for broader 2D) settings. Alongside this, co-designing network architectures that anticipate the dynamic tightening and loosening of equivariance constraints could produce models that converge even faster, require fewer samples, or require fewer extra parameters.
Finally, the dual variables learned by our algorithm encode where and when symmetry is most benefi- cial; exploiting these variables for adaptive pruning sparsification of non-evuivariant components could also lead to more efficient models.
This work aims to investigate how training can be improved for equivariant neural networks, offering benefits in areas such as drug discovery and materials science. We not foresee any negative societal consequences.
7 Conclusion
We introduced Adaptive Constrained Equivariance (ACE), a Nomotopy-inspired constrained equivari- ance training framework, that learns enforce equivariance layer by layer via modulation coeffi- cients 7. Theoretically, we show how our constraints can lead to equivariant models, deriving explicit bounds on the approximation on the equivariance emror for partially equivariant models obtained from inequality constraints. Across N-body dynamics, molecular regression, shape recognition and motion-forecasting datasets, ACE consistently improves accuracy, sample efficiency and convergence speed, while remaining robust to input dropout and requiring no ad hoc schedules or -ppecii penalties. These findings suggest that symmetry can be treated as a "negotiable" inductive bias that is tightened when it helps generalization and relaxed when optimization requires flexibility, offering practical middle ground between fully equivariant and unrestricted models. Extending the framework to other symmetry groups and modalities (such as 2D graphs and images), designing architectures that anticipate the constraint dynamics, and exploiting the dual variables for adaptive pruning of non-equivariant layers are natural next steps. Overall, we believe that allowing networks adjust their level of symmetry during training provides a solid basis for further progress in training equivariant networks.
for fully equivariant models obtained via equality constraints and
10

---

8 Acknowledgements
A. Manolache and M. Niepert acknowledge funding by Deutsche Forschungsgemeinschatt (DFG, German Research Foundation) under Germany's ' Excellence Strategy EXC 2075 - -30774001 the support by the Stuttgart Center for Simulation Science (SimTech), the International Max Planck Research School for Intelligent Systems (IMPRS-IS), and the support of the German Federal Ministry of Education and Research (BMBF) as part InnoPhase (funding code: 02NUK078). A. Manolache acknowledges funding by the EU Horizon project ELIAS (No. 101 120237). The work of L.F.O. Chamon is supported by the Agence Nationale de la Recherche (ANR) project ANR-25- NNR-25-CE23-3477- 01 as well chair from Hi!PARIS, funded in part by the ANR Al Cluster 2030 and ANR-22- CMAS-0002. A. Manolache is grateful to B. Luchian for insightful discussions on symmetries and equivariant functions.
References
[1] Taco Cohen and Max Welling, Group equivariant convolutional networks. In Proceedings of the 33rd International Conference on International Conference on Machine Learning Volume 48, ICML' 16, page 2990-2999, JMLR.org, 2016. 1
[2] Siamak Ravanbakhsh, Jeff Schneider, and Barnabás Póczos. Equivariance through parameter- sharing. In Doina Precup and Yee Whye Teh, editors, Proceedings the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2892-2901. PMLR, 06-11 Aug 2017. BtRL:ttts://ppoceediiog mlr. press/v700 ravanbakhshh/a.hnm htmll
[3] Taco S. Cohen and Max Welling. Steerable CNNs. In International Conference on Learning Representations, 2017. URL httpss:/ net iicrrmm?id-- rd-kyQKyt51111
[4] Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2757-2755, PMLR, 10-15 Jul 2018. URL https://proceedingg mlr press/ wttes/80//kooo ww0/kondor88aaaoomm html
[5] Daniel E. Worrall, Stephan J. Garbin, Daniyar Turmukhambetov, and Gabriel J. Brostow. Harmonic networks: Deep translation and rotation equivariance. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7168-7177, 2016. URL https
//api.semanticscholll. mrrpollarrrrggoo : 206596746
[6] Risi Kondor, Zhen Lin, and Shubhendu Trivedi. Clebsch- gordan nets: fully fourier space spherical convolutional neural network. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, page 10138-10147, Red Hook, NY, USA, 2018. Curran Associates Inc.
[7] Taco S. Cohen, Mario Geiger, Jonas Köhler, and Max Welling, Spherical CNNs. In International Conference on Learning Representations, 2018. URL httppttttps//eeeeeeeeeew http:pp:///peaereeeeeeet///orr net/forum?
Hdbbbd5xx2Rb
[8] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. International Conference on Learning Representations, 2017. [
[9] Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning, 2017.
g
[10] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018.
[11] Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference Learning Representations, 2019. [
11

---

[12] Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Klaus- Robert Müller. Schnet: continuous-fille convolutional neural network for modeling quantum interactions. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 991-1001, 2017.
[13] Nathaniel Thomas, Tess E. Smidt, Steven M. Kearnes, Lusann Yang. Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation- and translation equuiaaion neural networks for 3d point clouds. ArXiv, abs/1802.08219, 2018. URL httph:tt/s/est api. www.nticscc.iiieee.eee.....
org/CorpusID : 3457605
3 8
[14] J. Klicpera, F. Becker, and S. Günnemann. Gemnet: Universal directional graph neural networks
for molecules. CoRR, 2021.
[15] llee Batatia, David P Kovacs, Gregor Simm, Christoph Ortner, and Gabor Csanyi Mace: Higher order equivariant message passing neural networks for fast and accurate force fields. In Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 1423-11436 Curran Associates, Inc., 2022. URL httptt/ppp//ppppeeeeeeeeeeeeeeee neuriss.eppppp.p. cc/ wwwpppr..cpppppppppeeeepppeee_ee faweeeeffilleeepeeeeeeeeeeeeeeee pttpp/pppp////f/eeeeee 4a36c3c51af 1436G3351aaff1dfff441116666bbbbb Cantec---apppe----oeeeeeeee pdf
[16] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool. Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-589 2021.
[17] Viktor Zaverkin, Francesco Alesiani, Takashi Maruyama, Federico Errica, Henrik Christiansen, Makoto Takamoto, Nicolas Weber, and Mathias Niepert. Higher-rank irreducible cartesian tensors for equivariant message passing. In Conference Neural Information Processing Systems, 2024. [
[18] Ahmed Elhag, T. Rusch, Francesco Di Giovanni, and Michael Bronstein. Relaxed equivariance via multitask learning. 10 2024. doi: 11485500/aXxi.22111177777 2
[19] Mircea Petrache and Shubhendu Trivedi. Appooximaation-gnnraaaaaatooo trade-off's under (ap- proximate) group equivariance. In Thirty-seventth Conference on Neural Information Processing Systems, 2023. URL
[20] Nadav Dym and Haggai Maron. On the universality of rotation poinariaantooonntoooooooooo
point cloud networks.
In International Conference Learning Representations, 2021.
3 6
[21] Chaitanya K. Joshi, Cristian Bodnar, Simon Mathis, Taco Cohen, and Pietro Liò. the expressive power of geometric graph neural networks. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org. 2023.
[22] Yongjoo Back, Yariv Kafri, and Vivien Lecomte. Dynamical symmetry breaking and phase transitions in driven diffusive systems. Physical Review Letters, 118(3), January 2017. ISSN 1079-7114. doi: 10.1 11 PhysRevLett. 118. 030604
URL http:://xx oooi.org/ii 10. 1103/
[23] Simon A. Weidinger, Markus Heyl, Alessandro Silva, and Michael Knap. Dynamical quantum phase transitions in systems with continuous symmetry breaking. Phys. Rev. B, 96: 134313, Oct 2017. doi: 10. PhysRevB. 96. 134313
URL https://link: aps org//oi/10..o 1103/
[24] Calum J. Gibb, Jordan Hobbs, Diana I. Nikolova, Thomas Raistrick, Stuart R. Berrow, Alenka Mertelj, Natan Osterman, Nerea Sebastián, Helen F. Gleeson, and Richard. J. Mandle. Spontancous symmetry breaking in polar fluids. Nature Communications, 15(1), July 2024. ISSN 2041-1723. doi: 10. 10.00///44147--02-5000000 941467-024-5023--
URL.ttpp//dxx doi orw.oog/111 10. 1038/
[25] Tess Smidt, Mario Geiger, and Benjamin Miller. Finding symmetry breaking order pa- rameters with euclidean neural networks. Physical Review Research, 3, 01 2021. doi: 00.1033PyyyeevResarrr......1..
12

---

[26] Rui Wang, Robin Walters, and Rose Yu. Approximately equivariant networks for imperfectly symmetric dynamics. 01 2022. doi: 1448550/arxx.2221..111.
[27] Rui Wang, Elyssa Hofgard, Han Gao, Robin Walters, and Tess E. Smidt. Discovering symmetry breaking in physical systems with relaxed group convolution. In Proceedings of the 41
International Conference on Machine Learning, ICML'24. JMLR.org. 2024. D 6 25
[28] Andrew Blake and Andrew Zisserman. Visual reconstruction. MIT Press, Cambridge, MA, USA, 1987. ISBN 0262022710. |
[29] Alan Yuille. Energy functions for early vision and analog networks. Technical report, USA, 1987.
[30] Michael R. Osborne, Brett Presnell, and Berwin A. Turlach. A new approach to variable selection in least squares problems. Ima Journal of Numerical Analysis, 20:389-403, 2000. [
[31] D.M. Malioutov, M. Cetin, and A.S. Willsky. Homotopy continuation for sparse signal rep- resentation. In Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005., volume 5, pages v/733-v/736 Vol. 5, 2005. doi: 10.11000CASP:22000..4446.
[32] David L. Donoho and Yaakov Tsaig. Fast solution of noomm minimization problems when the solution may be sparse. IEEE Transactions on Information Theory, 54:4789-488 2008. [
[33]
Daniel Dunlavy and Dianne O'leary. Homotopy optimization methods for global optimization. 12 2005. doi: 10.2172/876373.
[34] Hidenori Iwakiri, Yuhang Wang. Shinji Ito, and Akiko Takeda. Single loop gaussian homotopy method for non-convex optimization. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL htttps://openreviexxx wtt..eeeeeeooommmddaa0007777111
[35] Xi Lin, Zhiyuan Yang. Xiaoyuan Zhang, and Qingfu Zhang, Continuation path learning for homotopy optimization. In Proceedings of the 40th International Conference on Machine Learning, ICML'23, JMLR.org, 2023.
[36] S.B. Gelfand and S.K. Mitter. Recursive stochastic algorithms for global optimization in ir/sup d/. In 29th IEEE Conference on Decision and Control, pages 220-221 vol.1, 1990. doi: 10.1109/CDC.1990.2000
I
[37] Maxim Raginsky, Alexander Rakhlin, and Matus Telgarsky. Non-convex learning via stochastic gradient langevin dynamics: nonasymptotic analysis. In Satyen Kale and Ohad Shamir, editors, Proceedings of the 2017 Conference on Learning Theory, volume 65 of Proceedings of Machine Learning Research, pages 1674-1703 PMLR, 07-10 Jul 2017. URL https
/pprocedding.mii mlr. wttp://igw.mm....e//e//eS3
[38] Alexandre Belloni, Tengyuan Liang, Hariharan Narayanan, and Alexander Rakhlin. Escaping the local minima via simulated annealing: Optimization of approximately convex functions. In Annual Conference Computational Learning Theory, 2015.
[39] S. Kirkpatrick, C. D. Gelatt, and M. Vecchi. Optimization by simulated annealing. Science. 22014598):671-6600 1983, doi: 10.1 10.1126/sceence 250.4498.7....44 URL https:/:::///e http::/wwwwwwciinne science org/doi/abs/ 1026.....11111 wit////sciennn snceeciieenccccccc 220 4498.6696 671
[40] L. Ingber. Simulated annealing: Practice versus theory. Mathematical and Com-
puter Modelling, 18(11):29-57 1993. ISSN 0895-7177. doi: https://doi.org/1...0 cww/ com///ciienc///rriieee
0895- 0895-7177793990244-C pii p11/0895717793390244C
URL bttts://www.scienceeee
[
[41] Stefanos Pertigkiozoolou, Evangelos Chatzipantazis, Shubhendu Trivedi, and Kostas Dani- ilidis. Improving equivariant model training via constraint relaxation. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 83497-83520.. Curran Associates. Inc., 2024. URL httptttp:://pooceedieee neurips..... cc/ wacc/pape__ffee files/ paper/20244 file/ 2BB oorpaaaaeeeeeeeeeeeaeeoooooooeee Conference pdf
10 30
13

---

[42] Fabian B. Fuchs, Daniel E. Worrall, Volker Fischer, and Max Welling. Se(3) 3c(3)-transformerss 3d
roto- translation equivariant attention networks. CoRR, abs/2006. 105003 2020. URL hhtpss
//arxiv. wwrxxivorrg/ab//2000 2006. 10503
g TO
[43] Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E(n) equivariant graph neural networks. CoRR, abs/2102.09844, 2021. URL httpss///axxi org www./ogg/////a/eee /atss//212...m 09844
[44] Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik J Bekkers, and Max Welling, Geometric and physical quantities improve e(3) equivariant message passing. In International Conference on Learning Representations, 2022. URL httpss//oppeereeieee www.reerrreieeeeeeeerreeer
id= x=__xx88@0BBe
28
[45] Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tägliasacchi, and Leonidas J. Guibas. Vector neurons: A general framework for -equivarra networks. In ICCV, pages 12180-12189, IEFE, 2021. ISBN 978-1-6654-2812-55 3
30
[46] Minkai Xu, Jiaqi Han, Aaron Lou, Jean Kossaifi, Arvind Ramanathan, Kamyar Azizzadenesheli, Jure Leskovec, Stefano Ermon, and Anima Anandkumar. Equivariant graph neural operator for modeling 3d dynamics. arXiv preprint arXiv:2401.11033 2024. 10
[47] Shuxiao Chen, Edgar Dobriban, and Jane H. Lee. A group-theoretic framework for data augmentation. Journal of Machine Learning Research, 21(245):1-71, 2020. URL http
//jmlr org/ papers/ Caapers/v21/200 w21/20-111120 163. html
[48] Alberto Bietti and Julien Mairal. Group invariance, stability to deformations, and complexity of deep convolutional representations. J. Mach. Learn. Res., 20(1):876-924, January 2019. ISSN 1532-4435.
[49] Akiyoshi Sannai and Masaaki Imaizumi. Improved generalization bound of group invariant / equivariant deep networks via quotient feature space. Uncertainty in Machine Learning, 2021.
[50] Mare Finzi, Gregory Benton, and Andrew Wilson. Residual pathway priors for soft equiv- ariance constraints. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 30037-
30049, Curran Associates, Inc., 2021. URL https: httpp:://proceeeee
neurips. ww/ppp/pp..ppp......p Paper .opprr.pffff
[51] Nate Gruver, Mare Anton Finzi, Micah Goldblum, and Andrew Gordon Wilson. The lie deriva- tive for measuring learned cquivariance. The Eleventh International Conference on Learning Representations, 2023. URL https::/openrreviez wttp::/wwwwwewtwwwwt......t wttp://////orrr/.--/..////a1/1
[52] Samuel E. Otto, Nicholas Zolman, Nathan Kutz, and Steven L. Brunton. unified framework to enforce, discover, and promote symmetry in machine learning, 2024. URL https://arx:: brgg/bss2311.000 00212
[53] Ignacio Hounie, Alejandro Ribeiro, and Luiz F. O. Chamon. Resilient constrained learning. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023, Curran Associates Inc. 5 6
[54] L. F. O. Chamon, A. Amice, S. Paternain, and A. Ribciro, Resilient control: Compromising to adapt. In IEEE Control and Decision Conference, 2020. 5
[55] L. F. O. Chamon and A. Ribeiro. Probably approximately correct constrained learning. In Conference Neural Information Processing Systems--(NuurIP)) 2020. 5
[56] I. F. O. Chamon, S. Paternain, M. Calvo-Fullana, A. Ribeiro. Constrained learning with non-convex losses. IEEE Trans. Inf. Theory, 1939):173--1760, 2023. doi: 109/TT.
1022.3187948.1 5
[57] Dimitri P. Bertsekas. Convex optimization theory. Athenea Scientific, 2009. 5
14

---

[58] Angelia Nedić and Asuman Ozdaglar. Approximate primal solutions and rate analysis for dual subgradient methods. SIAM Journal on Optimization, 19(4):1757-1780, 2009. doi: 10.1137/070708111 Utt.httss://doi. org/ 10.
wtt.:///////.111/77770/11111111 11370070708111
[59] Ashish Cherukuri, Enrique Mallada, and Jorge Cortes. Asymptotic convergence of constrained primal-dual dynamics. Systems & Control Letters, 87, 10 2015. doi: ..001//.sysconle22015..
10.006. S
[60] Roger A. Horn and Charles R. Johnson. Matrix analysis. Cambridge University Press, 2018.
[61] L. F. O. Chamon, S. Paternain, and A. Ribeiro. Counterfactual programming for optimal control.
In Learning for Dynamics and Control (L4DC), 2020. 6
[62] Jonas Köhler, Leon Klein, and Frank Noé. Equivariant flows: sampling configurations for multi-
body systems with symmetric energies, 2019. URL httts: http:///axvvvv..raabbbeee org/abs//9/a 00753
8
10
[63] Thomas N. Kipf, Ethan Fetaya, Kuan Chinh Wang, Max Welling, and Richard S. Zemel. Neural relational inference for interacting systems. In International Conference on Machine Learning,
pages 2693-27033 2018. 7
[64] L. Ruddigkeit, R. van Deursen, L. C. Blum, and J.-L. Reymond. Enumeration of 166 billion organic small molecules in the chemical universe database gdb-17. Journal of Chemical Information and Modeling, pages 2864-75, 2012. ACS.
[65] R. Ramakrishnan, O. Dral, P., M. Rupp. and O. Anatole von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data, 2014. Nature. .
[66] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In 2015 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 1912-1920, 2015. doi:
10.1 N0.11099CVPR.2215522988 7
[67] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon. Dynamic graph enn for learning point clouds. ACM Trans. Graph., 38(5), October 2019. ISSN 0730-0301. doi: 10.1 145/332636.. httphhttss///o..com
10. 1145/3326362
7
[68] Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and M. Solomon. Dynamic graph CNN for learning on point clouds. CoRR, 2018. 9
[69]
CMU. The cmu motion capture dataset, 2003. httpstts:///moaa cs cmu. cmu.edu/
g
[70] Weitao Du, He Zhang, Yuanqi Du, Qi Meng, Wei Chen, Bin Shao, and Tie- Liu. Equivariant vector field network for body system modeling. CoRR, 14s/2110.148111 2021. URL
https://arxiv. org/ abs//100/2211/.. 14811
I0
[71] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018. URR.hhttts://ppeereee
25
[72] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages
2019. 28
[73] Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning Graphs and Manifolds, 2019. 28
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, please see the "Homotopic Equivariant Training" and "Experimental Setup and Results" sections in the main text.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: Yes, please see the "Conclusions" section in the main text and the "Liimtations and Further Work" section in the Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer:
[Yes]
Justification:
Yes, proofs are provided in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Yes, detailed descriptions about the method, datasets, models and hyperparam- eters are provided in the "Experimental Setup and Results" section of the main text, and in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
17

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The datasets and base models are publicly available. The experimental setup details are in the main paper and in the Appendix. The code will be made publicly available upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.cc/public//////ccc heapiii/ggiis///Ce Caweae//ddoeSSbbbiica SorooddeumisssiooPPP.. wwriiisssiippcyioioo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The training splits, optimizer and hyperparameters follow previous literature. We detail the changes that we make in the Appendix. The code will be publicly released upon acceptance.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We perform multiple runs for our experiments (3-5) and provide mean results together with standard deviations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report runtimes and resourced used in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpp:://euuri:
Answer: [Yes]
Justification:
We reviewed the Code of Ethics and we respect it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss potential beneficial societal impacts in the Appendix section "Limitations and Further Work"" We could not identify potential for misuse or negative societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
19

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The datasets are publicly available and our models do not have a high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We do credit original owners of assets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, batasets.papersstt paperswi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our codebase will be made publicly available after acceptance. We will provide instructions for running the code upon release.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Paper does not involve crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No experiments involving crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
21

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs are used only for writing, editing, or formatting purposes and do not impact the core methodology, scientific rigorousness, originality of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
22

---

A Appendix Supplementary Materials
A.1 Pseudo-Code for ACE
To incorporate ACE, extend standard equivariant model with additional parameters "/ and non-equivariant layers. The code below illustrates how to implement the architecture accordingly:
class Model (nn. Module):
def ii.it_
....)
self ganmas Parameterriist
nn Paraaaaar (rr.meetrrrtooch.........).. for in (n_yeer_
1)
self f f_neqq nn. ModuleList[[
NoneqModule(...> for range (n, layers)
1)
self.f_eq nn. ModuleListt(
EgModule(....) for in range (n_layers) })
def forvard(...):
(...)
for i in range (Len (self gammas))
self fillil (x) gamma [i] self fi]eeq[]] (x)
(...)
Training requires introducing dual variables and a corresponding dual optimizer, while keeping the primal optimizer unchanged. The snippet below shows how add these components and update them during training:
gammas model. mooel.get_ggaaeee get_ggammss( pan_mpareeeters lambdas
corch.zerost (1, dtype=torch.fledt requires_grad-Palee squeeze for in gammas
dual pawameters......e extend (lambdas)
# If we want to use Algorithm 2, add: # us = Parameter (torch. zeros (1, requires.grad-True squeeze ())
for range(len (len (lambdas))
dual_ parameerreetrree.. (xxend
aptimizer_prrmal optim. Aww./SGD/et ptrameel.arammtersss ())
optimizer_dual optim Adam/SGD/ (aarpate
# Now, in the training routine: (...)
loas loss_ foss_s,(predd, y) dual_loss
slacks [gamma for gamma in ganmas]
for dual_vax, var, slack in zip(1ambdas, slacks): dual_loss var slack
# Explicit, but can also be obtained via autodiff for i, slack in (slackk))) lambdas [i] grad -glack
# If we use Algorithm thorithm 2, change to:
# loss loss 1/2 torch. nor..ooootttthhhhoooooooooo stack (us)) ** 2 dual loss
23

---

# slacks = [torch. norm (gamma) for gamma, in zip(gamma,, us)]
for dual_var, slack, in zip(lanbdas, slacks, us):
dual_ (dual_vay var slack) (u slack)
# for i, (slack, u) in enmerrate(2ip(slaaae us)):
# lambdas [i] ggad sslakk
lagrangian loss dual loas lagrangian.backuard() backward optimizer.stepi belimizer_dua.sstepe
If we use Algorithm 2, add: for lambda in lambdas:
# lambda. clamp_ (min=0)
(a) Test MSE (x10-") the N-Body dataset un- der different initializations of y. "N/A" indicates that ACE is used.
(b) Test MSE (x10-") 3) on the N-Body dataset for different dual learning rates T/-- "N/A" indicates that ACE not used. The primal learning rate fixed to 7p = 9 10
n
Vinit
Test MSE
Test MSE
N/A
N/A
6.64 0.14
8 x
10-4
1
4.97 + 0.01
N/A
6.64 + 0.14
8 x
10
0
5.39 + 0.33
1 X 10-5 5
6.10 + 0.25
8 X 10
5.47 +: 0.29
1 10
5.24 + 0.29
8 10
4.97 = 0.01
X 10
5.03 - 0.09
8 x 10-3
5.11 + 0.16
Table Ablations on the N-Body dataset. (a) Effect of initializing y. (b) Effect of varying the dual learning rate 7/d-
A.2 Discussion regarding gamma initialization
All experiments in the main text use Vini 1. as we did not find it necessary to run a separate hyperparameter search over this value. To illustrate the effect of initialization, we performed an ablation with SEGNN on the N-Body dataset where we compared Timi = against Timi (Table Ba).
As can be seen, both models using ACE obtain improved performance compared to removing ACE altogether (πa = N/A, Tinit N/A). However, initializing with Vinit = 1 leads to the lowest test error. We observe empirically that that zero initialization encourages faster convergence the early stages of training, but could bias the optimization toward solutions that closely adhere to the symmetry constraints, potentially limiting exploration. In contrast, initializing Tini gives us slower initial convergence but ultimately achieves better final accuracy.
A.3 Relationship between primal and dual learning rates
The dual optimization in ACE introduces learning rate na in addition to the primal learning rate 7/p We observe empirically that setting J) close to 7, leads to stable convergence across architectures and datasets. From practical standpoint, it is often sufficient to first tune 1), and then either set 7)a = 7p or choose slightly smaller value (e.g., Nd 8x10-4 when 7/ = 9x 901))4). While default choices for 7p (as in official implementations) tend perform well, we find that da is less sensitive and can be specified by simple proportional scaling. To quantify this effect, we conducted a sensitivity study with SEGNN on the N-Body dataset with 1000 samples. (bb the test MSE (mean + std) under different T/a values, including baseline without ACE ("N/A"). The best performance was consistently achieved when y)a R T/p- confirming that matching the dual and primal learning rates provides a robust default setting.
24

---

Square Squree
Square Reasigg
Square New- Naw- Symmette
Figure 8: C4 symmetry toy experiment with ACE (resilient constraints). We want to transform square to other shapes. The strictly equivariant baseline succeeds only when the target shares Ca symmetry (top row). Under ACE with resilient constraints, the learned coefficient y adapts: in the aligned case, y decreases towards zero, and the outputs match; the mismatched cases (second and third rows), increases, adding a missing signal that corrects the symmetry mis mis-specification. For visualization, outputs are min-max normalized per panel.
A.4 Correcting symmetry mis-specification with ACE
Motivated by analyses of symmetry breaking in 2D CNNs 27 we take different route do not use relaxed group convolutions; instead, we keep the convolution strictly Cy-equivariant and relax equivariance via ACE by adding non-equivariant branch f an equivariant C4 CNN feg.
We study three image-to-image mappings that differ in how they align with the imposed C2 symmetry: (i) gqquare-Squaanl (aligned target; a C4-symmetric pattemn, (ii) Square saauee-Rectanlle (mismatched (no global symmetry). We train
symmetry; target requires C2), and (iii) Squarre-Non-Symmeele a strictly equivariant baseline (feg "no ACE") and an ACE model fcr + f under resilient (inequality) constraints. In Figure we see ACE model output decomposed into fea (x:), the correction 7, (x)), and their sum. The baseline solves Square->Suarre but fails for the mis- specified cases. With ACE, the learned coefficient behaves desired: in Square-+Squan Y decays towards 0 and the outputs of the models are similar, indicating that the imposed symmetry matches 7 grows, obtaining non-equivariant
the task; in Square-<Rectand correction that recovers the desired outputs. A natural direction for future work is to evaluate ACE across modern 2D equivariant architectures on real datasets and quantifying accuracy-equivarian trade-offs.
and Monarre#Non-Symmene
A.5 Controlling the scale of the non-equivariant component.
The non-equivariant layer fleg. can in principle dominate the constraint Y if its output norm grows too large. To prevent this, we assume that each such layer is M-Lipschitz and bounded operator, i.e., (x:)] < B||ar|| for all inputs æ during training. This condition can be enforced by reparameterizing the weights, for example by using Spectral Normalization [7]] which guarantees
B <
25

---

From practical standpoint, we found that enforcing exact equivariance (Algorithm 1 did not require explicit use Spectral Normalization: during model selection, we checkpoint based on the validation performance of the equivariant projection Sø which naturally disregards the non- equivariant part. However, in the case of Resilient Constrained Learning (Algorithm 2). Spectral Normalization becomes important for reducing the equivariance error of S + f (see Fig. 4
This consequence of not being bounded if the weights of fa f are not reparameterized.
Hence, Spectral Normalization provides a simple and effective mechanism to control the scale of the non-equivariant branch and to reduce the equivariance error.
A.6 Proof of Theorem 4.1
Here, we prove the more refined bound below:
Theorem A.I. Consider far as in satisfying Assumption 7 Then,
(x), (foo))))))
h8+11
hnsl
BM--111a1 for all x e X.
The bound in Theorem 4.1 can be obtained directly from the above using the fact that 7 S 7.
Proof. The proof proceeds by bounding the error layer by layer. Indeed, begin by defining
2o = x and 2, (2--( joq (24-1) + 7/ (z5-1). for 1....
(4)
so that 2r, = (z)) Likewise, let
20 oq = and oq = 50 1o (2,21)) for i = I,.....,
(5)
so that 2T, = fo.0(x). Proceeding, the triangle equality to obtain
- || 2 S (=--1) + +h+ll (x---)
(6)
We bound (6) using the fact that l is M-LLipcchizz continuous and that S is bounded opera- tor (Ass. to write
(2 (2 $ 2 - 2 = M
hilllse (z+ |||| < 14s 1/1
Back in 6 we get
5, S M + h7s B 2 ||.
Noticing that the quantity of interest is in fact [| (x) fo.o(x)]l 1| 2 or, we can use 7 recursively to obtain
6r $ B _ M
Pixk 2k |||
(8)
1=1
using the fact that &n = To proceed, note from Assumption and the definition of the architecture in (3) that f8,, is M(1 + hrll--Lipschizz and z; Já,, f Hence, obtain
2
< M hyD) [=]]. for i = 1,.... L.
(9)
1=1
Using the fact that = g immediately becomes
(1 l75l)) ) |2
8, < BM
Im +
Iyn Σ
(10)
1=1
26

---

To obtain the bound in Theorem A.1 suffices it to use the relation between the arithmetic and the geometric mean (AM-GM inequality) obtain
k-1 I [11+))) < 1
k-1
hel
1:
Rearranging the terms yields the desired result.
A.7 Proof of Theorem
Once again we prove a more refined version Theorem 4.2 Theorem A.2. Consider f0,7 as in 3 satisfying Assumption and let C max max(B/M,
L-1
- fo.- ( (xx(
2 hall
(
1+
C
Σ
hsl
B² M 1|211
R=1
5th
for all x E and g E
The result in Theorem 4.2 is recovered by using the fact that 7/ S 5.
Proof. We proceed once again in layer-wise fashion using the definitions in (4) and
ig = px (g)æ and & -
(0 + Io (2.-), for I,...,
Hence, we can write the equivariance error the i-th layer as
4 (1)))))))) - | = pe(o)s,0 (z--1) + ( mo (z:-) -- (Zi-1) -aso ( (2x
Notice that, this notation, we seek a bound on eg independent of X and G.
Proceeding layer-by-layer, use the equivariance of fou, and the triangle inequality to bound 6; by
4 S 1 (p- (p- 10 (2)-1) hil p((o) ((0))))))))))))) - (E:0
(11)
Using the fact that poy. Lipschitz continuous and frey. is bounded operator (Assumption 1. we obtain
1omd (p--1(9) 5o (2)-1) M (ll:-1(0)2211 - 2--11 = M
| noo (2--1) Sm (2-1) B(B - ||
We can further expand the second bound using the fact that || || P5-17 + 11-12 S ||ă Ps-1 +| S
When combined into we obtain the bound
4 4 (M BB B2 ||
(12)
Applying 12 recursively, we obtain
L | j=
EL S 2B2
hvd
(M + hyl
||
(13)
where we used the fact that eo (g)x -oll 0.
To proceed, we once again use Assumption and and the definition of the architecture in so that 113 becomes
to obtain
E1. S 2B2
hxl
II ==+11
(M + 1y(
M + haD]
|0|||
27

---

Table 4: Parameter counts and runtimes for the equivariant baselines versus models trained with ACE. We relax equivariance during training with extra linear or MLP layers, operations that PyTorch executes efficiently, therefore training and validation are only marginally slower than the baselines. At inference we drop these non-equivariant layers, yielding virtually identical runtimes to the pure equivariant models. All experiments were performed on a RTX A5000 GPU with an Intel 19- 1900K CPU.
Model
#Parameters (Train)
Train time (s/epoch)
Val time (s/epoch)
SEGNN N-body (vanilla)
147,424
1.368
0.505
SEGNN N- body (constrained)
937,960
1.5 1.512:0000
0.527:10.04
SEGNN QM9 (vanilla)
1,033,525
118.40 20.108
11.152.:0003
SEGNN QM9 (constrained)
9,869,667
129.039=0.00
11. 11,402
EGNO MoCap (vanilla)
1,197,176
0.702
0.130
EGNO MoCap (constrained)
1,304,762
1.10622000
0.130
DGCNN-VN ModelNee40 (vanilla)
2,899,830
57. 07.806=0.500
5,954
DGCNN-VN ModelNect0 (constrained)
8,831,612
81.760-0.169
5.978
SchNet QM9 (vanilla)
127,553
10.389=0112
0.944
SchNet QM9 (constrained)
153,543
11.126=0.155
0.947
To simplify the bound, take C max(B/M, 1), so that can write
I.
B
]
R-1
EL S 2M 2M1
B
hwl
II j=
1
hall
I
lasl)
|||||
M
k=1
(14)
S -- 2M
hrd
II (1 + Chsl) ||=||
A=1
Using once again the relation between the arithmetic and the geometric mean (AM-GM inequality) yields
1-1
e1. 2
7kl
1
C
Ensl
ML ||
L
A.8 Additional plots for y and X
While the main text characterizes only the evolution of the modulation coefficients nr omits the corresoonding dual variables A₁ which serve as the Lagrange multipliers enforcing each equivariance constraint. Here, we present the full, detailed plots of 7/ and X; for ModelNet40 (Fig. 9 and Fig. TTP and CMU MoCap (Fig. Trig.[iaam and Fig. [2]. allowing simultaneous. inspection of the symmetry relaxation and the accompanying optimization "pressure". By tracing >; alongside %s one can observe how retained constraint violations accumulate multiplier weight, driving subsequent adjustments to 7 and thereby revealing the continuous "negotiation" between inductive bias and training flexibility instantiated by our primal-dual al algorithm.
A.9 Additional Experimental Details
All experiments are implemented in PyTorch 722 using each model's official code base and default hyperparameters as starting point. Because several repositories employed learning rates that proved sub-optimal, we performed a grid search for both the primal optimizer and, when applicable, the dual optimizer in our constrained formulation. After selecting the best values, we re-ran every configuration with multiple random seeds. noteworthy mention that the test results on the QM9 use different splits for the training, validation and test datasets. For
dataset reported in Table 2 SchNet, we shuffle the dataset obtained from PyTorch Geometric [73] and split the dataset with 80%/10% 10% vainnvaliiationn/est ratio. For SEGNN,
the official splits from [4].
28

---

500
300
623-
-15-
100
200
Figure 9: Full layer-wise Y values ModelNet40. The ACE VN-DGCNN quickly drives the modulation coefficients "/ of the first few layers towards O, locking those layers into fully equivariant behaviour, while deeper layers exhibit brief oscillations before convergence.
606
-00-
010
M
.208
Figure 10: Dual variables A; on Net4 The companion values of X mirror the % dynamics: layers whose 7/2 further away from zero accumulate larger Aç, signaling higher constraint cost.
w
wwwmmm
wwwmmm
-66
1.0
w
w
226
720
12:00 $200
3753
2000
250
200
1200
3800
12.00
30.00
1753
1250 2000
2000
Figure 11: Layer-wise % dynamics CMU MoCap (Subject #9, Run) dataset. Rather than decaying zero as on the MedelNet40 dataset, the modulation coefficients 7 oscillate around zero for the full 2000-step training window, signaling that imposing strict equivariance is overly restrictive for the dataset and model.
29

---

wmmmmm
mmmcomm
62-
1900 1276 3900
2000
1083
1993
1aho 1900 2030
Figure 12: The dual variables X; on CMU MoCap (Subject #9, Run). Similar to the % coefficients, the Lagrange multipliers X; oscilate near zero, but never converge.
In the case of VN-DGCNN, we followed the setup of Pertigkiozoglou et al. [41]. which trains cquivariant models on sub-sampled resolution of 300 points per point cloud. This differs from the original VN-DGCNN paper [5], where 1024 points are used. Moreover, our implementation only tuned the primal and dual learning rates (7p Ila), without modifying other components or performing a broader hyperparameter search. Due to these differences, both in input resolution and in the extent of tuning, the reported results are not directly comparable those Deng al. 445]
Table 5 lists the learning rates explored and the number of seeds per dataset.
The code and instructions on how to reproduce the experiments are publicly available at https 7/github. htt.ttuu...ommmaaaaaa aadrre///deeeeeeeaee imano/ACB ACE
Table 5: Learning rates and number of random seeds used for each dataset.
Dataset
Primal LR grid
Dual LR grid
Seeds
ModelNer40
{0.10, 0.20
[1x10-1, 1x10-3)
CMU MoCup
(5x10-9 1x10-3)
(5x10-5 2x 2x10- 1x
N- N-Body
10-10-1 8x 9x 10-1, 1x-0-3, 11110-3,000 1x100=,5x1003
[5x10 10 8x 10-
5
QM9
5x110 5x 1x10-
(5x10-
1x10-2 1xx10
1x
5
All of the experiments were performed on internal clusters that contain mix of Nvidia RTX A5000,
RTX 4090, or A100 GPUs and Intel i9-11900K, AMD EPYC 7742, and AMD EPYC 7302 CPUs.
All experiments consumed total of approximately 1000 GPU hours, with the longest compute being consumed on the QM9 dataset. Runtimes and parameter counts are reported Table 4
30


Reviews:
[{"review_id": "ars1VBbCYe", "reviewer": "Reviewer_bnrd", "summary": "This paper proposes a novel training framework that overcomes the limitations of training strictly equivariant models by treating equivariance as a soft constraint and utilizing techniques from the constraint optimization literature. Specifically, the proposed Adaptive Constraint Equivariant (ACE) framework augments the equivariant model with per-layer non-equivariant unconstrained layers for which the output is modulated by parameters $\\gamma$. The problem of training an equivariant network is then expressed as a constraint optimization problem, where the equivariant constraint is reduced to the equality constraint of $\\gamma=0$. The authors then propose to optimize the original problem by optimizing the dual problem via gradient descent on the model parameters and gradient ascent on the dual variables. Additionally, the authors provide supporting theoretical analysis of the approximation error that occurs when the network returns to the equivariant parametrization and the end of training, by setting $\\gamma=0$, and describe a mechanism that can allow users to infer possible symmetry misspecification by looking at the convergence of the $\\gamma$ variable. Finally, the experimental section provides empirical evidence demonstrating the benefits of the proposed ACE framework for a wide range of equivariant tasks.", "questions": "- What is the sensitivity of the optimization of the dual problem to the choice of learning rate for both the parameters of the model and the dual variables?\n- Is it possible that the non-equivariant component of the homotopic architecture increases the scale of its outputs during training as a way to counteract the reductions of the $\\gamma$ variable?\n- What is the reason for the gap between the reported performance of VN-DGCNN in this paper and the performance reported in the original Vector Neurons paper?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The authors propose a task-agnostic framework that, in contrast with previous works, doesn't rely on a user-specified learning schedule and allows for a data-dependent modulation of the equivariant constraint during training.\n- The framework can be applied to both equivarariant and approximate equivariant tasks.  Section 4.2 provides a nice heuristic for determining what symmetries are part of the tasks and whether a fully equivariant or approximately equivariant approach is more beneficial.\n- The formulation of the homotopic architectures requires minimal assumptions about the type of model to which the ACE framework can be applied.\n\nWeaknesses:\n- While the authors claim that their framework doesn't require additional user tuning, the optimization of the dual problem introduces additional hyperparameters (e.g. the learning rate for the dual variable $\\lambda$). There is no study of the sensitivity of the optimization to these additional parameters.\n- There is a gap between the accuracy reported for VN-DGCNN in Figure 7 and the accuracy reported in the original paper for ModelNet40. Additionally, in Figure 7, there is a small typo in the name of the model used.\n- In the homotopic architecture, the non-equivariant component is not constrained, and although it is a bounded operator by B, B is free to become arbitrarily large during training to counterbalance the reduction of $\\gamma$. This process can increase the approximation error shown in Theorem 4.1, even from small $\\gamma$ values.\n- Minor: There is an inconsistency in the naming of the different methods between Figure 2 and Figure 7. In Figure 2, the normal SEGNN (which is an equivariant model) is referred to as \"No Constraints\" model, while in Figure 7, the equivariant VN-DGCNN is referred to as \"Equivariant\" model.", "was_revised": true, "final_justification": "In the rebuttal the authors addressed all of the concerns I expressed in my initial review. I believe that this work both provides a useful methodology for training equivariant and approximate equivariant method while also provides useful insights about the nature of the problem. Due to the above I raise my rating to 5.", "rebuttal": "We would like to thank Reviewer bnrd for their thoughtful and constructive review. In what follows we clarify the Reviewer's concerns.\n\n**W1 - Learning rate $\\eta_d$ for the dual variables**: Both Reviewer bnrd and Reviewer bUHD point out that, while our method indeed doesn't require hand-crafted schedules for $\\gamma$ or tuning of regularization hyperparameters, the dual problem optimization involves a new learning rate $\\eta_d$. Empirically, we have observed that having a dual learning rate $\\eta_d \\approx \\eta_p$ works well in most situations. Our recommendation from a practical perspective is to find a good learning rate $\\eta_p$ and then set $\\eta_d = \\eta_p$, or to set $\\eta_d$ to a slightly lower value (e.g. if $\\eta_p = 9e^{-4}$ then set $\\eta_d=8e^{-4}$). We usually had a harder time searching for $\\eta_p$ than $\\eta_d$, as can be seen in Supplementary Table 4. Using the default learning rates for $\\eta_p$ (e.g., from the official implementation of a model) will generally produce good results, but we observed that in some cases they were suboptimal, as stated on Line 951 in the Supplementary.\n\nWe have run a sensitivity analysis on the N-Body dataset (1000 samples). Please note that the table contains a baseline experiment ($\\eta_d$=N/A means ACE is not used). The initial values for $\\eta_p, \\eta_d$ were obtained via the initial learning rate grid search (Table 4 of the Supplementary). We set $\\eta_p=9e^{-4}$ for all experiments and only evaluate with the strictly equivariant projection $f_{eq}$.\n\n| $\\eta_d$  | Test MSE ($\\times 10^{-3}$) |\n| --------- | --------------------------- |\n| N/A       | $6.64\\pm0.14$               |\n| $1e^{-5}$ | $6.10\\pm0.25$               |\n| $8e^{-5}$ | $5.47\\pm0.29$               |\n| $1e^{-4}$ | $5.24\\pm0.29$               |\n| $8e^{-4}$ | $4.97\\pm0.01$               |\n| $1e^{-3}$ | $5.03\\pm0.09$               |\n| $8e^{-3}$ | $5.11\\pm0.16$               |\n\nAs can be seen, different learning rates $\\eta_d$ can affect final performance. However, in all experiments, using ACE produced better results than the base model (N/A), with the best results being obtained when $\\eta_p \\approx \\eta_d$. \n\nWe agree with the reviewer that this is an important point. We believe that this discussion adds further insight to our method and is very useful from a practitioner's perspective. We will include the discussion together with additional plots in the next iteration of our paper.\n\n---\n\n**W2 - Performance gap between official VN-DGCNN and our method**: In our experiments, we adopt the setup from Pertigkiozoglou et al. [1], a related work that also focuses on training equivariant models. This setup uses a sub-sampled resolution of 300 points per point cloud, in contrast to the original VN-DGCNN paper [2], which uses 1024 points. Additionally, in our implementation, we only tuned the primal and dual learning rates $\\eta_p, \\eta_d$, without modifying other components or performing a broader hyperparameter search.\n\nDue to these differences, both in input resolution and in the level of tuning, our results are not directly comparable to those reported in [2]. We will continue to explore the source of these differences and include the additional results and a discussion in the final version of our paper.\n\nWhat is more, we thank the Reviewer for pointing out the typo in Figure 7.\n\n---\n\n**W3 - Having a large norm for the non-equivariant component might increase the scale of its outputs:** While it is indeed true that the magnitude of the output of the non-equivariant layer $f_\\theta^{neq,i}$ might dominate the constraint $\\gamma_i$, we assume that the layer is M-Lipschitz and a bounded operator with $||f_\\theta^{neq,i}||\\leq B||x||$ holding throughout training. This can be easily enforced by reparametrizing the weights, e.g., by using Spectral Normalization [3] (L180-181) so that $B\\leq 1$. \n\nFrom a practical standpoint, when seeking to obtain exact equivariance (Algorithm 1), we did not need to use Spectral Normalization --- when doing model selection, we checkpoint based on the validation performance of the equivariant projection $f_\\theta^{eq}$, ignoring the non-equivariant component. However, we have observed that in the case of Resilient Constrained Learning (Algorithm 2), Spectral Normalization is needed for the equivariance error of $f_\\theta^{eq} + f_\\theta^{neq}$ to decrease (Figure 4); this is indeed due to B not being bounded if the weights are not reparametrized. \n\nWe will add a more explicit discussion regarding the use of Spectral Normalization in the final version of our paper.\n\n---\n\n**W4 - Naming Inconsistencies:** We thank the Reviewer for pointing out the naming inconsistencies. We will fix them for the final version.\n\n---\n\nOnce again, we kindly thank the Reviewer for taking their time to review our paper. We believe that addressing their concerns has helped us significantly improve the clarity of our paper and allowed us to add some interesting discussions to it. We are open to further discussions in the next reviewing phase!\n\n---\n\n[1]: Pertigkiozoglou et al., \"Improving Equivariant Model Training via Constraint Relaxation\", NeurIPS 2024\n\n[2]: Deng et al, \"Vector Neurons: A General Framework for SO(3)-Equivariant Networks\", ICCV 2021\n\n[3]: Miyato et al, \"Spectral Normalization for Generative Adversarial Networks\", ICLR 2018"}, {"review_id": "YmaM4OHVhg", "reviewer": "Reviewer_hCpi", "summary": "It is well known that equivariant models cause complex loss functions even if the data obey symmetry and lower the expressiveness of the model. The authors propose Adaptive Constrained Equivariance: an optimization framework that starts from a non-equivariant model and introduces gradually equivariance via homotopy enabling thus a smooth transition from a non-equivariant to an equivariant model.", "questions": "Q1: L. 214 Not clear what it means by bounded translation. \n\nQ2: It is not clear what symmetry was assumed in the MoCap dataset. What is the homogeneous space and which is the transitive action? Is it a global SO(3)?\n\nQ3: While the gamma and loss plots give some insight, I think a very simple example with two weights and a very easy symmetry like SO(2) could give a very clear insight in the studied behaviors.", "limitations": "none identified.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "S1: The first ever optimization that automatically transitions from a free to an equivariant model.\n\nS2: Theoretical guarantees on the equivariance approximation.\n\nS3: It avoids manual tuning of losses and optimization schedules\n\nS4: The authors cleverly exploited the fact that dual methods in constrained optimization are instances of homotopy continuation. This is explained well by starting with the dual of the pure equivariant problem, namely the empirical Lagrangian, which approximates the solution of the equivariant problem. Algorithms 1 and 2 juxtaposed illuminates very well the difference.\n\nS5: Stopping the iteration over the non-equiv contribution $\\gamma$ is formally shown to produce a bounded equivariance error if both equivariant and non-equivariant models are Lipschitz continuous.\n\nS6: The authors moreover show that when data are partially summetric the non-equivariant model approximately satisfies the equivariance constraint with a proven bound. \n\nS7: The trade-off between performance and equivariance can be resolved with a resilient constrain loss based on slacks. \n\nS8: It is awesome that the authors formalize three questions/hypotheses for the experiments.", "was_revised": true, "final_justification": "One of the best papers I have reviewed for Neurips in the last few years. The responses to all reviewers are illumnating.", "rebuttal": "We're very happy to hear that Reviewer hCpi found our paper interesting and compelling!\n\nIn the following, we will address the questions raised by the Reviewer:\n\n**Q1 - Bounded Translation clarification**: We did not find a reference to “translation” or “bounded” on line 214 and therefore assume that the Reviewer refers to line 204. There, we highlight that in most cases, the transformations that we are interested in are isometries, i.e., norm-preserving transformations (B = 1), the most common of which are rotations and translations (involving either padding or wrapping). The results of Theorem 4.2, however, hold for arbitrary B-bounded transformations. We will add additional clarifications regarding this. \n\nWe might, however, have misunderstood the question of the Reviewer; if our answer is not satisfactory, we kindly ask the Reviewer to ask us for further clarification.\n\n---\n\n**Q2 - Symmetry assumed in MoCap**: The symmetry assumed for MoCap is the Special Euclidean group $SE(3)$ --- so translations $\\mu \\in \\mathbb{R}^3$ and rotation matrices $R\\in SO(3)$. Intuitively, as mentioned in the original EGNO paper [1], \"*As a practical example, to model dynamics by learning a function to predict $\\mathcal{G}^{(t+1)}$ from $\\mathcal{G}^{(t)}$, we hope that any rotations and translations applied to the current state coordinate $x(t)$ should be accordingly applied on the predicted next one $x(t+1)$, while the velocities $v$ should also rotate in the same way but remain unaffected by translations*\". We will add further clarifications in the final version of our paper.\n\n---\n\n**Q3 - Simple example with an easy symmetry**: We thank the Reviewer for this suggestion. As also noted by Reviewer k8VA, incorporating experiments with 2D equivariant convolutions would be a valuable addition to our paper.\n\nWe are currently preparing experiments inspired by the Relaxed Group Convolutions introduced in Wang et al., 2024 [2], where the authors visualize the weights of a relaxed $C_4$-equivariant convolution under different symmetry groups, such as $C_2$. In a similar spirit, we plan to train a $C_4$-equivariant network using ACE and analyze how the non-equivariant component $f_\\theta^{neq}$ changes across different symmetry settings.\n\nFurthermore, since $f_\\theta^{neq}$ can be interpreted as a correction term to $f_\\theta^{eq}$ in the presence of symmetry mismatch, we also plan to visualize and compare the contributions of $f_\\theta^{eq}(x)$ and $\\gamma f_\\theta^{neq}(x)$ to the final prediction. These visualizations will help illustrate how ACE adapts when exact symmetry is not present.\n\n---\n\nOnce again, we would like to kindly thank the Reviewer for their overwhelmingly positive assessment of our paper and for their feedback. If they have any further questions, we would be happy to address them!\n\n---\n\n[1]:  Xu et al., \"Equivariant Graph Neural Operator for Modeling 3D Dynamics\", ICML 2024\n\n[2]: Wang et al., \"Discovering Symmetry Breaking in Physical Systems with Relaxed Group Convolution\", ICML 2024"}, {"review_id": "u9ramFyfvc", "reviewer": "Reviewer_ZRDi", "summary": "This paper introduces a novel technique for training approximately equivariant networks. Specifically, the networks are assumed to have an architecture consisting of a linear combination of equivariant layers with non-equivariant layers, where the weights of the latter are learned through optimization to control the degree of non-equivariance. In particular, the authors' principal contribution is a specialized training method based off of the principles of homotropy via interpretation of the objective as a generalized Lagrangian.", "questions": "The key question I have is regarding the implementation. It seems like the application of Algorithm 2) would require a modification of a standard optmizer (such as Adam/AdamW).  Perhaps I missed it, but it appears that this is not discussed in the paper. \n\nIs it necessary to modify the optimizer to handle the proposed training method? If not that is a big benefit and should be stated. If not, then a detailed explanation of a general recipe should be stated in the main paper or appendix, in addition to the release of the authors code.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "I'm a fan of this paper. The authors contributions are well scoped, well explained, and supported by rigorous theoretical and extensive experimental results. In particular, the latter show broad applicability across a wide variety of tasks and architectures.   I think moving away from strictly equivariant models is important, as it is becoming clear that such models are ill-suited for practical applications where symmetries are often partial, messy, or otherwise corrupted.\n\nOverall I think this paper will be a nice contribution to the conference and I am happy to recommend clear acceptance. That said, I have some minor questions/concerns below.", "was_revised": true, "final_justification": "I had very minor concerns coming into the rebuttal, which the authors fully addressed. \n\nI think this is a strong paper about an important topic, and a necessary step for bridging the gap between equivariant theory and real-world performance. \n\nI think this paper is a clear accept, and will be happy to advocate for it.", "rebuttal": "We would like to thank the Reviewer for reviewing our work and for their positive assessment!\n\nIn the following, we address their concern regarding the implementation of the primal-dual algorithm:\n\nFor both the strictly equivariant version (Algorithm 1) and the partially equivariant Resilient Constrained Learning version (Algorithm 2), only minimal modifications are required. In practice, implementing a new optimizer is not necessary. In our codebase, we use the standard PyTorch implementations of SGD or Adam (though any optimizer can be used for either the primal or dual updates) and define separate optimizers for the primal and dual variables. The same approach can be easily adapted to other frameworks such as TensorFlow and JAX.\n\n---\n\nIn the following, we provide a practical PyTorch example showing how a standard equivariant architecture with a conventional optimization routine can be adapted to use ACE:  \n\n1. Modifications to the model - adding the $\\gamma$ and the non-equivariant parameters:\n\n\n```\nclass Model(nn.Module):\n    def __init__(...):\n        (...)\n        self.gammas = nn.ParameterList([nn.Parameter(torch.tensor(1.)) for _ in range(n_layers)])\n        self.f_neq = nn.ModuleList([NoneqModule(...) for _ in range(n_layers)])\n        self.f_eq = nn.ModuleList([EqModule(...) for _ in range(n_layers)])\n\n    def forward(...):\n        (...)\n        for i in range(len(self.gammas)):\n            x = self.f_eq[i](x) + gamma[i] * self.f_neq[i](x)\n        (...)\n\n```\n\n  \n\n2. Modifications to the training routine:\n```\ngammas = model.get_gammas()\ndual_parameters = []\nlambdas = [torch.zeros(1, dtype=torch.float, requires_grad=False).squeeze() for _ in gammas]\ndual_parameters.extend(lambdas)\n\n# If we want to use Algorithm 2, add:\n# us = [nn.Parameter(torch.zeros(1, requires_grad=True).squeeze()) for _ in range(len(lambdas))]\n# dual_parameters.extend(us)\n\noptimizer_primal = optim.Adam/SGD/etc(model.parameters())\noptimizer_dual = optim.Adam/SGD/etc(dual_parameters)\n\n# Now, in the training routine:\n(...)\nloss = loss_fn(preds, y)\ndual_loss = 0\nslacks = [gamma for gamma in gammas]\n\nfor dual_var, slack in zip(lambdas, slacks):\n    dual_loss += dual_var * slack\n\nfor i, slack in enumerate(slacks): # Explicit, but can be obtained via autodiff\n    lambdas[i].grad = -slack\n\n# If we use Algorithm 2, change to:\n# loss = loss + 1/2 * torch.norm(us) ** 2\n# dual_loss = 0\n# slacks = [torch.norm(gamma) - u for gamma, u in zip(gammas, us)]\n# for dual_var, slack, u in zip(lambdas, slacks, us):\n#    dual_loss += (dual_var * slack) - (u * slack)\n# for i, slack, u in enumerate(zip(slacks, us)):\n#    lambdas[i].grad = -slack - u\n\nlagrangian = loss + dual_loss\nlagrangian.backward()\noptimizer.step()\noptimizer_dual.step()\n\n# If we use Algorithm 2, add:  \n# for lambda in lambdas:\n#     lambda.clamp_(min=0)\n\n```\n\nWe will include this example on implementing ACE in the Supplementary material of the final version of the paper, along with links to the official implementation.\n\n---\n\nWe would once again thank the Reviewer for their feedback and are open to address any further questions or concerns that they might have!"}, {"review_id": "K7oS7reVOl", "reviewer": "Reviewer_k8VA", "summary": "This paper introduces Adaptive Constrained Equivariance (ACE), a framework for handling approximately equivariant problems. Each layer is decomposed into an equivariant block plus a non-equivariant branch scaled by $\\gamma$. Training starts with $\\gamma=1$ (fully flexible) and gradually shrinking $\\gamma$ through the constrained optimization procedure, smoothly interpolating from non-equivariant to strictly equivariant behavior. Experiments on N-body dynamics, molecular regression, shape recognition, and motion-forecasting tasks demonstrate the effectiveness of ACE.", "questions": "1. Would the authors consider adding a 2-D convolutional experiment that directly compares ACE with [26, 27]?\n2. The current experimental section feels crowded. Splitting it into clear subsections (for example, by data domain) would improve readability.\n2. For each task, could the authors explicitly state whether the ground-truth function is strictly or only approximately equivariant? Highlighting this distinction would help readers interpret the results.\n4. Do all experiments initialize $\\gamma$ with 1? Have the authors considered different initial $\\gamma$ values, such as starting with $\\gamma=0$?", "limitations": "Although the authors discuss limitations, the section is relegated to the appendix. Moving it into the main text in the final version would improve transparency and balance.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n1. The methodology is generic and can be plugged into diverse backbone architectures to tackle approximately equivariant settings.\n2. The approach is thoroughly evaluated across multiple domains and base architectures, showcasing broad applicability.\n\nWeakness\n1. As the authors note, there are no experiments on 2-D convolutional networks. Given the wide usage of (equivariant) CNNs, such a study would greatly strengthen the paper.\n2. The paper compares against only one relaxed-equivariance baseline. Including additional methods such as [26, 27, 50] would make the empirical comparison more convincing.", "was_revised": true, "final_justification": "The authors rebuttal resolved most of my questions, and it is great to hear that they are working on the 2D conv experiment. So I keep my initial score of accept.", "rebuttal": "We would like to thank Reviewer k8VA for their thoughtful and constructive feedback. In the following, we address each of their concerns in detail.\n\n**Q1 - ACE with 2D convolutions:**  \n\nWe thank the Reviewer for this suggestion. As also noted by Reviewer hCpi, incorporating experiments with 2D equivariant convolutions would be a valuable addition to our paper.\n\nWe are currently preparing experiments inspired by the Relaxed Group Convolutions introduced in Wang et al., 2024 [27], where the authors visualize the weights of a relaxed $C_4$-equivariant convolution under different symmetry groups, such as $C_2$. In a similar spirit, we plan to train a $C_4$-equivariant network using ACE and analyze how the non-equivariant component $f_\\theta^{neq}$ changes across different symmetry settings.\n\nFurthermore, since $f_\\theta^{neq}$ can be interpreted as a correction term to $f_\\theta^{eq}$ in the presence of symmetry mismatch, we also plan to visualize and compare the contributions of $f_\\theta^{eq}(x)$ and $\\gamma f_\\theta^{neq}(x)$ to the final prediction. These visualizations will help illustrate how ACE adapts when exact symmetry is not present.\n\n---\n\n**Q2 - Experimental Section is too crowded:**  \n\nThe Reviewer has a good point. The final version of the paper will include one additional content page that we will use  to refactor the experimental section for clarity and structure. Specifically, we will reorganize the section into subsections, expand on the descriptions of each experimental setup, and provide more detailed analysis of the results.\n\n---\n\n**Q3 - Obtaining Partially vs. Exactly Equivariant Functions; Exact Symmetries Across Experiments:**  \n\nWe appreciate the Reviewer’s observation and will revise the paper to clarify the distinction between exact and partial equivariance. In particular, we will specify for each experiment which symmetry assumptions are enforced exactly and which are only approximately satisfied.\n\n---\n\n**Q4 - Initializing $\\gamma$ at Different Values:**  \n\nAll experiments use $\\gamma_{init}=1$ (we did not see the need to run any hyperparameter search for $\\gamma$). Below, we show the results of an experiment on the N-Body dataset that shows the effect of using $\\gamma_{init}=0$:\n\n| $\\eta_d$  | $\\gamma_{init}$ | Test MSE ($\\times 10^{-3}$) |\n| --------- | --------------- | --------------------------- |\n| N/A       | N/A             | $6.64\\pm0.14$               |\n| $8e^{-4}$ | 1               | $4.97\\pm0.01$               |\n| $8e^{-4}$ | 0               | $5.39\\pm0.33$               |\n  \nAs can be seen, both cases yield better results than not using ACE ($\\eta_d=N/A, \\gamma_{init}=N/A$). However, better performance is obtained when starting in an unconstrained scenario ($\\gamma_{init}=1$).\n\nThese results indicate that a zero initialization may lead to faster convergence, but it may also bias the model toward solutions that closely follow the enforced symmetries, potentially limiting exploration in the early stages of training. In contrast, initializing $\\gamma_{init}=1$ results in slower convergence, but can lead to better final performance. Due to this year's NeurIPS policy restricting the use of images in the rebuttal, we are unable to include optimization dynamics plots at this stage. However, we will include these plots along with a detailed discussion in the Appendix of the final version.\n\n---\n\n**Limitations Section:**  \n\nFor the final version, we will move the limitations discussion to a dedicated section before the conclusions.\n\n---\n\nOnce again, we would like to kindly thank the reviewer for taking their time to review our work and provide us with constructive feedback. We believe that addressing these points has helped us significantly improve the clarity of our paper and allowed us to add some important discussions to it.\n\n---\n\n[27]: Wang et al., \"Discovering Symmetry Breaking in Physical Systems with Relaxed Group Convolution\", ICML 2024"}, {"review_id": "9YO0xhu18z", "reviewer": "Reviewer_bUHD", "summary": "The paper introduces Adaptive Constrained Equivariance (ACE), a constrained optimization framework that applies layer-wise homotopy for equivariance training. The key novelty is leveraging the dual problem and tighten the constraints via dual descent. The paper proposed algorithms for strictly and partially equivariant data, and provide theoretical analysis on the optimality bound. A range of experiments demonstrate the superior performance of the proposed framework, as well as analysis on training dynamics and comparison with theoretical bounds.", "questions": "1. The ACE framework introduces new step size parameters $\\eta_p$, $\\eta_d$. How sensitive is the method to these parameters, and what are some general rules for setting these values in practice?\n2. Assumption 1 is fairly strong, for example ReLU networks are non-differentiable. So in real-data scenarios where the assumptions are violated, which aspects of the framework will be most vulnerable to failure? How robust is the framework to such violations?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strength\n1. The paper studies the well-motivated problem of equivariant network training, and provides novel insight with the proposed dual problem formulation.\n2. The theorems provide concrete tight error bounds on equivariance performance.\n3. Experimental evidence is rich and covers performance evaluation, model analysis and alignment with theoretical results.\n4. The proposed framework is applicable to general network architectures.\n5. Organization and presentation of the paper is clear.\n\nWeakness\n1. Assumption 1 is strong in practice, and having some discussions on its connection with real scenarios will be helpful.\n2. The newly introduced step size parameters $\\eta$ still needs tuning.", "was_revised": true, "final_justification": "I appreciate the authors' response, and the explanation on choices of the $\\eta$ parameters. Such rule of thumb results can be helpful, while in practice these parameters may still need to be tuned. Overall I will maintain my score.", "rebuttal": "We would like to thank Reviewer bUHD for their comments and for the positive assessment of our work. In the following, we address the Reviewer’s concerns:\n\n**W1 - Learning rate $\\eta_d$ for the dual variables**: Both Reviewer bUHD and Reviewer bnrd point out that, while our method indeed doesn't require hand-crafted schedules for $\\gamma$ or tuning of regularization hyperparameters, the dual problem optimization involves a new learning rate $\\eta_d$. Empirically, we have observed that having a dual learning rate $\\eta_d \\approx \\eta_p$ works well in most situations. Our recommendation from a practical perspective is to find a good learning rate $\\eta_p$ and then set $\\eta_d = \\eta_p$, or to set $\\eta_d$ to a slightly lower value (e.g. if $\\eta_p = 9e^{-4}$ then set $\\eta_d=8e^{-4}$). We usually had a harder time searching for $\\eta_p$ than $\\eta_d$, as can be seen in Supplementary Table 4. Using the default learning rates for $\\eta_p$ (e.g., from the official implementation of a model) will generally produce good results, but we observed that in some cases they were suboptimal, as stated on Line 951 in the Supplementary.\n\nWe have run a sensitivity analysis on the N-Body dataset (1000 samples). Please note that the table contains a baseline experiment ($\\eta_d$=N/A means ACE is not used). The initial values for $\\eta_p, \\eta_d$ were obtained via the initial learning rate grid search (Table 4 of the Supplementary). We set $\\eta_p=9e^{-4}$ for all experiments and only evaluate with the strictly equivariant projection $f_{eq}$.\n\n| $\\eta_d$  | Test MSE ($\\times 10^{-3}$) |\n| --------- | --------------------------- |\n| N/A       | $6.64\\pm0.14$               |\n| $1e^{-5}$ | $6.10\\pm0.25$               |\n| $8e^{-5}$ | $5.47\\pm0.29$               |\n| $1e^{-4}$ | $5.24\\pm0.29$               |\n| $8e^{-4}$ | $4.97\\pm0.01$               |\n| $1e^{-3}$ | $5.03\\pm0.09$               |\n| $8e^{-3}$ | $5.11\\pm0.16$               |\n\nAs can be seen, different learning rates $\\eta_d$ can affect final performance. However, in all experiments, using ACE produced better results than the base model (N/A), with the best results being obtained when $\\eta_p \\approx \\eta_d$. \n\nWe agree with the reviewer that this is an important point. We believe that this discussion adds further insight to our method and is very useful from a practitioner's perspective. We will include the discussion together with additional plots in the next iteration of our paper.\n\n---\n\n**W2 - Assumption 1 is strong:** The main reason why we are making Assumption 1 is that the magnitude of the output of the non-equivariant layer $f_\\theta^{neq,i}$ might otherwise dominate $\\gamma_i$. \n\nWe would argue that the assumption is not strong, and can indeed be easily satisfied in practice - we assume that the layers are M-Lipschitz and bounded operators with $||f_\\theta^{neq,i}||\\leq B||x||$. This can be easily enforced by reparametrizing the weights, e.g., by using Spectral Normalization [1] (L180-181) so that $B\\leq 1$, even when using ReLU non-linearities. Moreover, we **do** use ReLU non-linearities in practice when $f_\\theta^{neq}$ is a MLP.\n\nFrom a practical standpoint, when seeking to obtain exact equivariance (Algorithm 1), we did not need to use Spectral Normalization --- when doing model selection, we checkpoint based on the validation performance of the equivariant projection $f_\\theta^{eq}$, ignoring the non-equivariant component. However, we have observed that in the case of Resilient Constrained Learning (Algorithm 2), Spectral Normalization is needed for the equivariance error of $f_\\theta^{eq} + f_\\theta^{neq}$ to decrease (Figure 4); this is indeed due to B not being bounded if the weights are not reparametrized. \n\nWe will add a more explicit discussion regarding the use of Spectral Normalization in the final version of our paper.\n\n---\n\nOnce again, we would like to kindly thank the Reviewer for taking their time to review our paper and provide constructive feedback. We believe that these points have allowed us to improve both the practical aspects and the theoretical clarity of our method. We are happy to discuss any further questions or concerns in the next stage.\n\n---\n\n[1]: Miyato et al, \"Spectral Normalization for Generative Adversarial Networks\", ICLR 2018"}]

Author rebuttal:

Meta-review: The paper presents a novel method for optimizing equivariant neural networks by initially utilizing a non-equivariant model and gradually transforming it to become increasingly equivariant. This approach cleverly applies homotopy methods from optimization. All the reviewers appreciated this innovative strategy and awarded high scores. I am pleased to recommend acceptance for an oral presentation.
Decision: Accept (oral)