Improving Equivariant Model Training via Constraint Relaxation
Authors: Stefanos Pertigkiozoglou
Evangelos Chatzipantazis
Shubhendu Trivedi
Kostas Daniilidis
Venue: NeurIPS 2024 poster (2024)
Keywords: Equivariant Neural Networks
Symmetries
Approximate Equivariance
Optimization
Primary area: optimization_for_deep_networks
TL;DR: Improving the optimization of equivariant neural networks by relaxing the equivariant constraint during training

Abstract: Equivariant neural networks have been widely used in a variety of applications due to their ability to generalize well in tasks where the underlying data symmetries are known. Despite their successes, such networks can be difficult to optimize and require careful hyperparameter tuning to train successfully. In this work, we propose a novel framework for improving the optimization of such models by relaxing the hard equivariance constraint during training: We relax the equivariance constraint of the network's intermediate layers by introducing an additional non-equivariant term that we progressively constrain until we arrive at an equivariant solution. By controlling the magnitude of the activation of the additional relaxation term, we allow the model to optimize over a larger hypothesis space containing approximate equivariant networks and converge back to an equivariant solution at the end of training. We provide experimental results on different state-of-the-art network architectures, demonstrating how this training framework can result in equivariant models with improved generalization performance. Our code is available at https://github.com/StefanosPert/Equivariant_Optimization_CR

Improving Equivariant Model Training via Constraint Relaxation
Stefanos Pertigkiozoglon University of Pennsylvania pstefano@seas. upenn. edu
Evangelos Chatzipantazis' University of Pennsylvania vaghat@seas upenn. edu
Shubhendu Trivedi
Independent
shubhendu@csail. mit edu
Kostas Daniilidis University of Pennsylvania Archimedes, Athena RC kostas@cis upenn. edu
Abstract
Equivariant neural networks have been widely used in a variety of applications due their ability to generalize well in tasks where the underlying data symmetries are known. Despite their successes, such networks can be difficult to optimize and require careful hyperparameter tuning to train successfully. In this work, propose novel framework for improving the optimization of such models by relaxing the hard equivariance constraint during training: We relax the equivariance constraint of the network's 's intermediate layers by introducing an additional non- equivariant term that we progressively constrain until we arrive at an equivariant solution. By controlling the magnitude of the activation of the additional relaxation term, we allow the model to optimize over a larger hypothesis space containing approximate equivariant networks and converge back at the end of training. We provide experimental results on different state-of-the art network architectures, demonstrating how this training framework result in equivariant models with improved generalization performance. Our code is available http:://jiibbbb.ommeeeeeeeeeeerp CatprriannnnpppmiiiiiiioC_CCC
equivariant solution
1 Introduction
The explicit incorporation of task-specific symmetry in the design and implementation of effective
and parameter-efficient neural network (NN) models has matured into a rational and attractive NN design meta- porta-formalimm in recent years years-that that of group equivariant convolutional neural networks
(GCNNs) (Cohen & Welling 2016 6 Ravanbakhsh et al. 2017 Esteves et al. 2018 Kondor & Trivedi
2018, Cohen al.]] 2019 Maron al. [eiler Weiler Cesa 2019 Bekkers 2020 Villar et al.) [021]
Xu et al. 2022 Searce-Crump 2023). GCNNs involve using the machinery of group and representa- tion theory to compose layers that are equivariant transformations of the input. Such networks, with hard-coded symmetry. have proven to be successful across a wide variety of tasks, while often affording significant data efficiency. Such tasks/domains include: RNA structure (Townshend et
202I), protein structure (Back et al., [222 Jumper et al., (022), molecule generation (Satorras et al.
2021), medical imaging (Winkels & Cohen 2019), natural language processing (Gordon et al. 2020
Petrache & Trivedi 2024), computer vision (Chatzipantazis et al. (0233). robotics (Zhu al. 2022
Ordoñez-Apracz et al. 2024), density functional theory (Gong et al.) (023), particle physics (Bo
gatskly et all, 2020), lattice gauge theories (Boyda al. 2021) amongst many others. GCNNs
now have also matured enough to have a well-developed theory. This includes both prescriptive (or
Equual Contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

architectural) theory and descriptive analysis. In general, GCNNs particularly stand out in domains with data scareity, or with high degree of symmetry (Kufel et al. 2023 Boyda al. 202I). or the physical sciences where respecting explicit symmetries could be dictated by physical laws, violating
which could lead to physically implausible predictions.
Despite the successes of group equivariant models, there are several outstanding challenges that don't yet have general satisfactory solutions. We discuss two that have attracted recent attention. The first challenge- primary motivation of our paper-has to do with the common observation that equivariant networks can be difficult to train (Wang et al. 2024, Kondor et al., 2018 Liao & Smidt, 2023). The reasons for this general difficulty are not well -ell-undersood, but it seems to occur in part because the training dynamics of such networks can be notably different from nou-equivariant architectures. For instance. if a GCNN operates entirely in Fourier space (Bogatskiy et al. 2020 Kondor et al., 2018: Xu et al., 2022). most of the usual intuition about training NN models does not apply. Further, depending on the level of equivariance error tolerance for task, the internal layers could be computationally intensive, and involve e.g- higher-order tensor products. Notably, the above difficulty arises even when the model is correctly specified i.e. the model and the data encode the same symmetry. The second challenge with GCNNs, has to do with the downsides of working with exact equivariance when the data itself might have some (possibly) relaxed symmetry. This has recently led to a spurt of work on developing more flexible networks that can vary the amount of equivariance depending on the task (Finzi et al. 202 Romero & Lohit, 2022 van der Ouderaa et ala 2022] Waan et al. [0222] Huang et al. [023] Petrache & Trivedi (023). Such models generally improve accuracy and will sometimes also simplify the optimization process as side-effect. Broadly, proposed solutions involve adding additional regularization terms that penalize for relaxation errors,
solving for the problem of model mis-specification (Petrache & Trivedi 2023).
However, even though there is now work on relaxed eqqivariant networks that addresses model mis- specification, existing works don't 't focus on improving the optimization process directly. In this paper, we take a step towards examining this question in more detail. We make the case that even if we assume that the model is correctly specified, relaxing the equivariance constraint during optimization and then projecting back to the equivariant space can itself help in improving performance. We conjecture that prime reason for the optimization difficulty of GCNNs, compared to non-equivariant models, is that their solution-space might be too severely constrained. We derive regularization terms that encourage each layer operate in larger hypothesis space during training- being constrained to only be in the intertwiner space-whill while encouraging equivariant solutions. After the optimization is complete, we project the solution back onto the space of equivariant solutions. The approach can also be adapted to better optimize approximately equivariant networks in a similar manner. The focus of our work thus distinguishes it from works on relaxed equivariance- we are not concerned with mis-specification, but rather with isolating the optimization process itself.
Below summarize the main contributions of our work:
We present novel training framework that can improve the performance of equivariant
neural networks by relaxing the equivariance constraint during training and projecting back to the space of equivariant models during testing (as shown in Figure 17.
We present a formulation that extends existing equivariant neural network architectures to be approximately equivariant. We show how training on the relaxed network can improve the performance of its equivariant subnetwork.
We provide experimental evidence showcasing how our framework improves the perfor- mance of existing state-of-the-ar equivariant architectures.
2
Related Work
There is little prior work on providing general procedures for improving the optimization process for equivariant neural networks directly. [Elesedy& & Zaidi (202T) sketched projected gradient method to construct equivariant networks and suggested a regularization scheme that could be used to implement approximate equivariance. However, this was proposed as an aside in the paper (sections 7.2 and 7.3), without empirical theoretical backing. Our work also involves projected gradient procedure. However, the regularization scheme that we propose is substantially different.
We use "relaxed" to encompass notions of partial and approximate equivariance Petrache & Trivedi 2023

---

Work on approximate and partial equivariance (Finzi et al. 2021 Romero & Lohit 2022, van der
Ouderaa et arl 2022, Wang et al. 2022; Huang et al., 2023; Petrache & Trivedi [023] Wang et al. 2023) seems superficially related to ours, but comes with different motivation. Such methods aim to match data symmetry with model symmetry and are not explicitly concerned with improving optimization. As a result, they are designed to address tasks with either inherent relaxed symmetries or tasks where the underlying relaxed symmetry is misspecified. Contrary to that, our method focuses on cases where the underlying symmetry is known exactly, and the relaxation of the equivariant constraint is used only during training as a way to improve the optimization. The works oi et al. (202T);
van der Ouderaa al. (2022) y Gruver al. (002);: : Otto et al. (024)) Petrache & Trivedi (2023) are
nonetheless relevant since they provide methods for measuring relaxed equivariance, comprising of
regularization schemes that are related to those used in our paper, since we also need measures of relaxation. In fact, the work of Gruver et al. (2023) directly inspires one component of our method. On the theoretical side, (etrachee & Trivedi (023) studied ammeralization-approiinaa approximately/fur
tradeoff's in equivariant CNNs in a very general setting, characterizing the effect of model mis-specification on performance. They quantify equivariance as improving the generalization error, and the alignment of data and model symmetries as improving the approximation error. They leave the impact of improving the optimization error for future work. While we do not provide theoretical results, our work could be seen as focusing on optimization error component of the classical picture
Maile et al. 2023) proposed what they call equivariance relaxation morphism, which reparamter- izes an equivariant layer to operate with equivariance constraints on a subgroup, but with the goal of architecture search. FFintth & Ohlsson (023) provide analysis of the optimization dynamics of equivariant models and compare them to non-equivariant models fed with augmented data. However, they don't 't use the analysis provide insights improving the optimization procedure itself.
Several researchers have recently tried to circumvent optimization difficulties in other ways. For canonicalization functions on top of large pre-trained models. The work of [Basu et al. (2023b) operates with a similar motivation but without canonicalization. Yet another representative of work with a fine-tuning motivation, but in a different context is (Basu et al.] 2023a). Finally, simplifying equivariant networks with heavy equivariant layers and improving their scalability is an active area of work and is related to easing optimization. Such works usually employ tools from representation theory, tensor algebra, or exploit sampling theorems over compact groups and their homogeneous spaces, such as Passaro & Zitnick (2023): Luo et al. (202)); Cobb et al. (202T): Ocampo et al. (2023).
instance. Mondal insdal et al. (2023) suggests using sntivariaacc-promoolie
3 Method
To introduce our proposed optimization framework, we first clearly define the equivariant constraint that the models aim train must satisfy. Assume a function R Rm and group G acting on the input and output spaces via the general linear representations Pmn : G - GL(R") )
Pout G + (RR)). Then the function is said to be equivariant to the action of group G if for all
g E G it satisfies the following constraint:
f(mn(g)))) = Powr (z)) for all x E 1
Assuming we use a neural network to approximate the function above, the definition of equivariance as stated doesn't 't impose specific constraints the individual layers of the network. Nevertheless, most of the current sate-of-the-an equivariant architectures are a composition of simpler layers each one of which is constrained to be equivariant. In this case, the overall model is the result of composition f = fN o fN-1 o... f2 o fi of simpler equivariant layers fa V/2 + V4+1, where V,, V the input and output spaces which the group G acts with the corresponding representations PV., Pv.+, (assuming V1= = R", VN= = IR' are the input and output spaces respectively).
In this work we focus family of models described above-that are defined through compo- sition of simpler equivariant linear layers. During standard training the linear layers are optimized over the set of intertwiners H,, i.e. the set of linear maps between the representations (Vr, Py;) and (Vi+, that have the equivariance property as stated in Equation T The set of intertwiners only subset of the set of all possible linear maps from Vi to Vi+1, and as result, they have a reduced
1 Characterizing model performance as generalization error + approximation enror + optimization error We assume henceforth that are always dealing with Matrix Lie groups
3

---

Standard Training
Training via Constraint Relaxation
Train and Test on Constrained Equivariant Space
relaxed Approximate Equivariant Space
Project to Equivariant Space during Testing
Equivariant Layer
Equivariant Layer
Linear Layer
Equivariant Layer
Linear Leyer
Equivariant
Equivariant Layer
Linear Layer
Equivariant Layer
Linear Layar
Layer
Figure 1: Standard training of equivariant NNs is constrained to a limited parameter space which result in challenging training process. We propose to relax these equivariant constraints during training, allowing optimization over a broader space of approximately equivariant models. During testing, project the trained model back the constrained ppace-arriving at an equivariant model with enhanced performance compared to equivalent models trained with the standard process.
number free parameters. We propose to facilitate training over this constrained space by relaxing the constraint imposed on the intermediate layers and optimizing over a larger hypothesis space H which a superset of the set of equivariant models H C H. trivial approach is to completely relax the constraint and solely optimize over the larger set containing all models. The problem with such an approach is that it completely abandons the concept of equivariance and all the attendant generalization benefits. Consequently, to expand the hypothesis space while keeping the benefits of equivariant models, we need a relaxation such that:
Given a non-equivariant model E Å,, we can efficiently return to an equivariant one E H.
The relaxed model has a small equivariance error P.e = Ez-p(>) Jc P (g) f(pn(g)x) [dg. |dg. This implies that although we extend the space models we optimize over, we do not diverge too far away from the space of equivariant solutions.
After we project back to the equivariant space, the error of the projection Ppe Ez<p(z) [x] [(+)]]] is also small. This ensures that while we optimize the less constrained model, can return the equivariant one without sacrificing the overall performance.
The first objective can be satisfied by defining intermediate layer of the form:
f(x) = (x) oWx, 0 2 0 and fe E H, W E V
(2)
where H is the set containing all possible equivariant solutions. Here is easy to see that we can O. which we refer to as projection to the equivariant
return to an cquivariant model by setting space. The formulation of the linear layer above is similar to the one used in the Residual Pathway Priors (RPP) (Finzi et al., 202I). Note that RPP, the value of / remains constant and acts as a prior on the level of equivariance we aim to control the value in order to actively change the level of equivariance during training and project back to the equivariance space during inference.
expect from given task and dataset. Contrasted to that, in this work
For the second objective, we need to utilize metric that measures the relative distance of the model from the space of equivariant models H. It was observed by Gruver et al. (2023) that an easy way to measure how much a model satisfies the equivariant constraints is by using the norm of the Lie derivative. We present details in the next section.

---

3.1 Lie Derivative Regularization Term
Assume we are given a matrix Lie group acting on vector space through its representation p: G - GL(V). For the given group there exists a corresponding Lie algebra with the property that for g. e^ E G. Additionally, there exists a corresponding Lie algebra representation dp: g g((V) such that p(e edo(A)e
we take the derivative of the action of a group element e EG at = we get the Lie derivative:
d
d
A
dd((A) dd((A)
dt
dt t=
Assume that the following group representation act on the vector space of functions as:
Pu-ouu(g)]ff (g) -1 (f (m)
As observed by Gruver et al. (2023) the lie derivative of the above action is zero for all equivariant
functions f. since for all g E the action Piu-out [g) f the identity map. As a consequence. we can use the norm of the Lie derivative as a metric to compute how much function f deviates from the equivariant constraint of Equation For the linear relaxation term Wz that we introduced in Equation 2
we have that the Lie derivative can be computed in straightforward manner as:
d
Ca(W)=
Pourle A (ie
dponl (A)t W (Pin(A)e
= dt 1==
= dt
1==0
= (A) + (A)
As a result, we can measure the degree that linear layer satisfies the equivariant constraint at point r, by computing the norm of the Lie derivative that point for each one of the generators of the group. For example in the case where G is the group of 3D rotations (G = (3))), we can compute the Lie derivative for each generator:
0
0
0
0 0
0 1 J=
J
1
Jo
0
0
0
-1
0/
0
During training, given an input distribution p(xx), we compute, for each linear layer x, the Lie derivative regularization term:
Cia(W) = Emer(H)
(W)x| (J)J:)
Σ
Although the above regularization applies when the symmetry group Lie Group, as we show in the experiments of Section (4.4] we can also define a similar regularizer for the case of discrete finite groups. In such case, for given linear layer with weights W and input x, we compute the sum of the norms of the difference Lg (W) jær (p(g,) W W W(gg)) for all generators 9j of the discrete finite group under consideration.
considering is matrix
As discussed in Otto al. (2024) and shown in Figure 3[a] the inclusion of the above regularization terms encourages equivariant solutions and prevents the model from diverging away from the space of cquivariant models. Moreover, Figure shows how the inclusion of this regularization helps the overall training and results in performance improvement of the final trained model.
3.2 Reducing the Projection Error
While we optimize over a larger hypothesis space, we always aim to return to an equivariant model after the end of training. Using the parametrization in Eq. we can always do that by setting 0 to be equal to zero. Although after the projection the resulting model is guaranteed to be equivariant, might be far from the original relaxed version, meaning it might have a large projection error Pr pe- Specifically, for an individual relaxed layer the projection error is:
Ppe Ex-p(a) [[][(x)) [(+)]]] Ezwp(*) (l) + +OWx fc(x)]l
Ex-p(z) [|]]
5

---

As result, to ensure that pee remains low we introduce second regularization term on the norm ||>l|] Additionally, reduce the contribution of on the projection error, we propose to schedule its value by slowly decreasing it during the last phase of training. Specifically, we apply a cyclic scheduling where given Ne total number of epochs, the value of 0 at epoch is:
2i
if N
8,
NE
i N
Ng
In Figure we show how both the additional regularization term on the norm of the activation ||a*]] and the scheduling of 0, affect the performance of our framework.
3.3 Training Objective
Overall, given a task with a corresponding loss Lass and data distribution D, our framework optimizes over the following training objective:
N
C = BlayyvD (f + Areg
(1)1--1(0)|++ [IG,(W.)5--1())
(5)
AEJ,
where W; is the weight matrix of the ith additive unconstrained linear layer and fi-1(x) is the output of the (é - 1)th layer (with fo corresponding to the input).
During training, we control the amplitude of the additive relaxation term by scheduling @ as described Section 3.2 During inference, we evaluate only on the equivariant part the model by setting 0 0. Thus as shown in Figure 1 after the end of training, the resulting model has the same parameter count and model architecture as the baseline model without any additional additive layers.
3.4 Relaxing the constraints of different equivariant architectures
In this section, consider selection of different equivariant architectures, and use them to illustrate how we could apply our proposed optimization framework:
Vector Neurons (Dngg ef al., 2021) In Vector Neurons, the primary linear layer processes features of the form X E E Nx3 It achieves equivariance by applying left multiplication with a weight matrix f(X) = WX. This operation mixes only the rows of the input feature matrix and as result when the input features rotate by R, the output also rotates since (X W W f(X)
To apply our proposed relaxation we add a linear layer that allows the mixing of all the elements of the input feature matrix. We achieve this by unrolling the feature matrix into vector of dimension (nm) and then after applying an unconstrainted linear layer, roll it back to a feature matrix. So the overall relaxed layer has the form:
f(x) W.X Ouvec [W [eec())
where vec, uvec are the corresponding unrolling and rolling operations.
SEGNN (Brandstetter et al. 2021) and Equiformer (Liao Smidt, 2023):
The intermediate
representation of both SEGNN and Equiformer are steerable vector spaces that transform according representation of SO(3). In particular, both models process a collection of type / tensors that transform according to the Wigner-D matrices ( (g) of the corresponding type l. The interaction between tensors of different types can be done using the Clebsch-Gordan (CG) tensor product which a bilinear operator that combines two input vectors 2/1, a/2 of types I and la and returns a tensor (ata @ ) of type I as follows:
4
la
(l.m) (l (la.m)) my (a(a))
my
mig
where xm. Zma are the mih m elements of tensors x! (!1)), (1a) and (l..)) ((.mm))0..m)) are the corresponding CG coefficients. In this operation, the CG coefficients restrict the possible interaction

---

between elements of different types of vectors. We relax the equivariant constraint by adding an unconstrained linear layer that can mix the elements from all the tensors used as intermediate representations, independent of their type. In Equiformer we add such a linear layer in the feed- forward network of the transformer block. Similarly in SEGNN, we add to the layer that receives the aggregated messages from all the neighbors of node and updates the node features.
Approximately Equivariant Steerable Convolutions (Wang et al., 2022): In this work, the authors designed approximate equivariant steerable convolutional layers. We apply our method by incorporating an additional unconstrained convolutional kernel. Since this task contains discrete symmetry groups, namely discrete rotations and scalings, we replace the Lie derivative regularizer with the corresponding one for discrete groups, described in Section
4 Experiments
4.1
Equivariant Point Cloud Classification
We first evaluate our optimization framework by training different networks on the task of point cloud classification. We use the equivariant variants of PointNet (Qi et al. 2016) and DGCNN (Wang et al.
2019) which were proposed by Deng et al. (2021). We train on the ModelNet40 dataset (Chang et al.
2015 which contains 12311 I point clouds from 40 different classes. We compare with the standard
Training of these networks using the same hyperparameter configuration as employed in Deng et al.)
22021)
During both training and testing, we sub- sub-sample the input point clouds to 300 points.
To apply our method we relax the Vector Neurons linear layer by following the methodology described in Section 3.4 For both networks we set the regularization term Xreg = 0.01, which is a value use in all of the following experiments. We provide more detailed description of the training parameters in Appendix [A.T] Furthermore, in Appendix [A.2 we describe the process of choosing the hyperparameter Xreg and show the method's robustness to its value. Figure 2 showcases how applying our proposed framework benefits the training of both networks. Specifically, for the case of the smaller and less performant PointNet, we can see an even larger performance increase over the baseline. These results show how the performance benefits of our optimization framework increase in smaller pnder-parametrized networks, an effect we investigate further in Section 4.2 In Appendix A..3 we provide additional details on the computational and memory overhead of our proposed optimization, showcasing that while additional parameters are introduced during training the overhead in the training time is limited.
Ablations on the regularization terms and scheduling: In addition to the training curves of our shoows the accuracy of our proposed optimization procedure
method and of the baseline, when we remove some of the proposed regularization terms or the scheduling of O. We observe that without any regularization both models diverge from the space of equivariant solutions. As a result, during inference when 0 their projection error Ppe becomes larger, resulting in a significant drop in test accuracy. Similarly, without the Lie derivative regularizer, the final test accuracy of both network variants drops. In such cases, Wx is unconstrained and can learn to extract cou-evuiva features that the equivariant part fe is not able to learn in any stage of the training. This effect can also be observed in figure ((a))shi showing the total Lie derivative of the network when is trained with and without the lie derivative regularization term. Not including the Lie derivative regularization allows the network, especially in the beginning of training, to optimize over solutions with large equivariance error. Finally, for both networks, we observe that f scheduling, as described in Section 3.2 can benefit training compared to fixing 0 to a constant low value. In Appendix A.4 we provide additional results sho showcasing how contrary to our method model with a constant 0 (without 0 scheduling) has a significant drop in performance after is projected into the equivariant space.
4.2 Scaling on Different Model and Dataset Sizes
To better understand how the model and dataset sizes affect our proposed optimization framework, we train models of variable depth on different numbers of training samples. As a baseline model we use the Steerable B(3) GNN (SEGNN) (Brandstetter et al. 2021) and we train on the task of Nbody particle simulation (Kipf et al. 2018). This task consists of predicting the position of 5 electrically charged particles after 1000 time steps when given as input their initial positions, velocities, and charges.
7

---

Equivariant Projected PN-notnt
Equivariant projected VN-DGCNN
06
Test Accuracy
Test Accuracy
Ours
Ours (no Theta Scheduling)
Ours (no Lie Deriv. Reg.)
04
Ours (no Regularization)
Original VNN
0.2
(no Thela Scheduling)
02
0.1
(no Derivative Reg.)
(no Regularization)
Original
0.-
100
150
Epoch
Epoch
Figure 2: Test accuracy on ModeINet40 classification, during training of equivariant PointNet and DGCNN using a baseline training process and different versions of our method. The accuracy is computed for the equivariant models, i.e. for the models after they are projected in the equivariant space.
Equivariant Frror during Training
Per Layer Distibution of Lie Deriv regularizer
Norm of Total Lie Derivative
Lie Desiv Regularization
Lic Derivative Regularization
8c
Layer
without Lie Denv. Regulanization
Layer
&
Layer
Layer
20
300 Epoch
Epoch
Figure 3: (a) Norm of the total Lie derivative of the relaxed PointNet model trained with and without the Lie derivative regularization term. For the computation of the Lic derivative we use the method proposed in Gruver al. (2023). (b) Value of the Lie derivative regularization term for each individual layer of the relaxed PointNet model while we train using our framework and with Lie derivative regularization weight Areg = 0.01
Figure ((a) shows the mean average test error achieved by networks of different sizes, both when trained with a standard optimization, and when trained with our proposed framework. We can observe that for all sizes our method achieves better generalizations. The gap between our method and the baseline becomes greater in the cases of smaller networks, phenomenon that also observed in the point cloud classification experiments in Section 4.1 Thus, our framework, by relaxing the constraint and introducing additional degrees of freedom, can help the overall optimization, especially in models with a limited number of parameters. Additionally. figure 4(b) shows that when we fix the model size and increase the dataset size our method able scale better than the baseline. In both cases, we can observe that the training of the baseline has a much larger variance and is highly dependent on the random initialization of the layers. On the contrary, method results in more consistent training with smaller variance between the random trials.
4.3 Molecular Dynamics Simulation
To evaluate our framework in challenging task using complex network architecture, we train Equiformer (Liao & Smidi (203) on the task of molecular dynamics simulations for set of molecules provided as part of the MD17 dataset (Chmiela et al. 2017). The goal of this task is to predict energy and forces from different configurations of pre-specified molecule. Following Liao Lia & Smidi (2023), for each molecule we use only 950 different configurations for training which significantly increases the task difficulty. For all training runs the same value of Xreg 0.01 in the

---

Nbody System Error (vs Model Size)
Nbody System Error (vs Dataset Size)
Mean Average Error (10 -)
SFGNN+Ours
SFGNN+Ours
SEGNN
Mean Average Error (10
SEGNN
30
10
20
2.5
500
40
5.0
Message Passing Layers
Training Samples (10*)
(a)
(b)
Figure 4: Mean Average Error on the Nbody particle simulation for (a) different model sizes, (b): different dataset sizes.
previous experiments and for the rest of the hyperparameters, we use the same configuration as the one proposed in Liao & Smidi (2023). In Table we show that the mean absolute error of energy and force prediction achieved by Equiformer, both when it is trained using standard training, and when it is trained with proposed optimization framework. Without any additional hyperparameter tuning, our framework able to provide improvements on the performance of Equiformer even for this challenging data-scarce task.
Table I: MAE of Equiformer trained with and without our optimization framework on a set of molecules from the MD17 dataset. The energy is reported in meV and the force in meV/À units
Aspirin
Benzene
Ethanol
Salicylic acid
Methods
Energy
Forces
Energy
Forces
Energy
Forces
Energy
Forces
Equiformer
5.3
7.2
2.2
6.6
2.2
3.1
4.5
4.1
Equiformer+Ours
5.2
7.1
2.2
6.6
2.0
2.9
4.1
4.1
4.4
Optimizing Approximately Equivariant Networks
Finally, we show how our framework can be beneficial, not only for the optimization of exactly cquivariant networks, but also for approximate equivariant ones. We apply our method on top of the approximate equivariant stecrable CNNs proposed in Wang et (2022). Although these models are not projected back to the equivariant space, they are still regularized stay within solutions with small equivariant етrоr. The main difference with our framework is that in the approximate equivariant setting, the equivariant relaxation remains the same throughout training. On the contrary, we propose to progressively constrain the model by modulating the value of the unconstrained term by slowly decreasing the value of 9 from Equation 2 As result by applying our optimization framework on top of the standard training of the approximate equivariant kernels, we test how progressively introducing additional constraints throughout training can help the performance of the network.
We evaluate our method on the task of 2D smoke flow prediction described in Wang et al, (2022). The inputs of the model are sequences of successive 64 64 crops of smoke simulation generated by PhiFlow (Holl et al. 2020). The desired output is prediction of the velocity field for the next time step. We evaluate two different settings: the "Future" setting where we evaluate on the same part of the simulation but we predict future time steps not included in the training, and the "Domain" setting where we evaluate on the same time steps as training but in different spatial locations. The data are collected from simulations with different inflow positions and buoyant forces. For the rotational symmetry case, while the direction the inflow and the buoyant force is symmetric to 90" degrees rotations (C4 symmetry group), the buoyancy factor changes for the different directions making the task not symmetric. For the scaling symmetry, the simulations are generated with different spatial and temporal steps, with the buoyant factor changing actoss scales.

---

Table 2: RMSE error on the synthetic smoke plume dataset with approximate rotational and scale symmetries. In the "Future" evaluation we train and evaluate the models the same simulation location but we test for later time steps in the simulation from the ones used in training. In the "Domain" evaluation we train and evaluate the models on the same timesteps but on different spatial locations in the simulation.
Model
MLP
Conv
Equiv
Rpp
Lift
RSteer
RSteer+Ours
Rotation Domuin
Future
1.38 0.06
1.21 0.01
1.05 = 0.06
0.96 1.9600.10
0.82 0.08
0.80 0.00
0.79 + 0.01
1.34 0.03
1,10 +.10+0000
0.76 + 0.02
0
0.68 0.09
0.67 0.01
0.58 1 0.00
Future
2.40 0.02
0.801
0.75 4 0.03
0.81 0.09
0.85
0.70 0.00
0.62 + 0.02
Scale
Domain
1.81 0.18
0.95 + 0.02
0.87 + 0.02
U.86 0.05
0.77 + 0.02
0.78 + 0.01
0.67 + 0.01
In addition the approximate equivariant steerable CNNs (RSteer), compare with simple MLP, with non-equivariant convolutional network (ConvNet), as well as with an equivariant convolutional network (Equiv) (Weiler & Cesa 2019) and with two additional approximate equivariant networks RPP (Finzi et al. 2021)) and LIFT (Wang et al. 2021) that are trained using a standard training procedure. In Table 2 we see that by applying our optimization framework the resulting approximate equivariant model outperforms all other baselines in both cases of approximate rotational and scale symmetry. These results indicate that starting from an unconstrained model and progressively increasing the applied constraints can benefit optimization even in the case where at the end of training we stay in the space of approximate equivariant models and do not project back to the cquivariant space.
5
Conclusion
In this work, we focus on the optimization of equivariant NNs. We proposed a framework for improving the overall optimization of such networks by relaxing the equivariance constraint and optimizing over a larger space of approximately equivariant models. We showcase the importance of utilizing regularization during training to ensure that the relaxed models stay close the space of equivariant solutions. After training, we project back to the equivariant space arriving at a model that respects the data symmetries, while retaining its high performance the task. We evaluate our proposed framework and its individual components over variety of different equivariant network architectures and training tasks, and we report that can consistently provide performance benefits over the standard training procedure, A theoretical analysis of our approach, possibly with appeal to empirical process theory (Pollard) 1990) to control the optimization error, is left for future work.
Acknowledgements
We gratefully acknowledge support by the following grants: NSF FRR 2220868, NSF IIS-RI 2212433, Support ARO MURÍ W911NF-20-1-0080, and ONR N00014-22-1-2677..
References
Baek, M., Dimaio, F., Anishchenko, I. V., Dauparas, J., Ovchinnikov, S., Lee, G. R., Wang, J., Cong, Q. Kinch, L. N., Schaeffer, R. D., Millán, C., Park, H., Adams, C., Glassman, C. R., DeGiovanni, A. M., Pereira, J. H., Rodrigues, A. V., van Dijk, A. A., Ebrecht, A. C., Opperman, D. J. Sagmeister, T., Buhlheller, C., Borkov-Keller T., Rathinaswamy, M. K., Dalwadi, U., Yip. C. K., Burke, J. E., Garcia, K. C., Grishin, N. V., Adams, P. D., Read, R. J., and Baker, D. Accurate prediction of protein structures and interactions using a three-track neural network. Science, 373: 871 - 876, 771, 2021.
Basu. S., Katdare, P., Sattigeri, P., Chenthamarakshan V., Campbell, K. D., Das, P., and Varshney, L. R. Efficient equivariant transfer learning from pretrained models. In Neural Information Processing Systems, 2023a. https:/:::://pp www.etiicccoooll.eoo
prg/Corpus ID: 258770888
Basu, S., Sattigeri, P., Ramamurthy, K. N., Chenthamarakshan, V., Varshney, K. R., Varshney, L. R., and Das, P. Equi-tuning: Group equivariant fine-tuning of pretrained models. In Williams, B., Chen, Y., and Neville, (eds.), Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth
AAAI Conference on Artificial Intelligence, AAAI 2023,
10

---

Symposiun on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023, pp. 6788 6796. AAAI Press, 2023b. doi: 10.1609/AAALL33716225522 URL https: //o.///o..// org/ 10. 1609/ aaai. v3716. 23716.25832
Bekkers, E. J. B-spline cnns on lie groups. In 8th International Conference Learning Repre- sentations, ICLR 2020, Addis Abubu, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
sttps://openreview..ntt www.:eeeffoormie hitoorrmmmiiiiaaaaae gBhkkFF
Bogatskiy, A., Anderson, B., Offermann, J., Roussi, M., Miller, D., and Kondor, R. Lorentz group equivariant neural network for particle physics. In International Conference on Machine Learning. pp. 111999-119 1002. PMLR, 2020.
Boyda, D., Kanwar, G., Racanière, S., Rezende, D. Albergo, M. S., Cranmer, K., Hackett, D. C., and Shanahan, P. E. Sampling using su (n) gauge equivariant flows. Physical Review D, 103(7): 074504, 2021.
Brandstetter, J., Hesselink, R., van der Pol, Bekkers, E., and Welling, M. Geometric and physical quantities improve e(3) equivariant message passing. 2021.
Chang, A. X., Funkhouser, T., Guibas, L.,, Hanrahan, P., Huang, Q., Li, Z.. Savarese, S., Savva, M., Song, S., Su, H., Xiao, J., Yi, L.. and Yu, F. ShapeNet: An Information-Rich 3D Model Repository. Technical Report arXiv: 1.1: 15111.01220022 [cs.GR], Stanford University - Princeton University Toyota Technological Institute Chicago, 2015.
Chatzipantazis, E., Pertigkiozoglou, S., Dobriban, E., and Daniilidis, K. Simathrm (3)) equivariant attention networks for shape reconstruction in function space. In The Eleventh In- ternational Conference on Learning Representations, 2023. URL https: /tttps://openrviiee net/ torum/1d= 10-K71111-
Chmiela, S., Tkatchenko, A., Sauceda, H. E., Poltavsky, I,, Schütt, K. T., and Müller, K.-R. Machine learning of accurate energy- cnergy-conserving molecular force fields. Science Advances, 3(5) 2017. doi: 10.1 (0.1126/sciadv.11603 sciadv 1603015
URL hhtpt:ttp//.. http://wwwwwccinccccce www.::/ciencccccccm www.e/eeee.....ooooeee/oooooooot http/doi//abbsst 1126/
Cobb, O., Wallis, C. G. R., Mavor-Parker, A. N., Marignier, A., Price, M. A., d' d'Avezac, M., and McEwwn, Efficient generalized spherical [enn)s. In International Conference Learning Representations, 2021. URL https://opppeeeeeeeee wtt.://oowrm22ter22233333335155
Cohen, T. and Welling, M. Group equivariant convolutional networks. In ICML,
Cohen, T. S., Geiger, M., and Weiler, M. A general theory of equivariant CNNs on homogeneous spaces. In Wallach, H., Larochelle, H., Beygelzimer, A., dAlché Buc, F., Fox, E., and Garnett, (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Deng, C., Litany, O., Duan, Y., Poulenard, A., Tagliasacchi, A., and Guibas, Vector neurons: a general framework for so(3)-equivariant networks. arXiv preprint arXiv:2104.12229 2021.
Elesedy, B. and Zaidi, S. Provably strict generalisation benefit for equivariant models. In International conference on machine learning, pp. 2959 2969. PMLR, 2021.
Esteves, C., Allen-Blanchette, C., Makadia, A., and Daniliddis, K. Learning so (3) equivariant representations with spherical cnns. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 52-68, 2018
Finzi, M., Benton, G., and Wilson, A. G. Residual pathway priors for soft equivariance constraints. In Advances Neural Information Processing Systems, volume 34, 2021.
Flinth, and Ohlsson, F. Optimization dynamics of equivariant and augmented neural networks, 2023.
Gong, X., Li, Zou, N., Xu, R., Duan, W., and Xu, Y. General framework for (3)-equivr neural network representation of density functional theory hamiltonian. Nature Communications, 14(1):2848, 2023.

---

Gordon, J., Lopez-Paz, D., Baroni, M., and Bouchacourt, D. Permutation equivariant models for compositional generalization in language. In ICLR, 2020.
Gruver, N., Finzi, M. A., Goldblum, M., and Wilson, A. G. The lie derivative for measuring learned equivariance. In The Eleventh International Conference on Learning Representations, 2023, URL https://openrevier nwt. cent/fooummidde id-
Holl, P. M., Um, K., and Thuerey, N. phiflow: A differentiable pde solving framework for deep learning via physical simulations. In Workshop on Differentiable Vision, Graphics, and Physics in Machine Learning at NeurIPS, 2020.
Huang, N., Levie, R., and Villar, S. Approximately equivariant graph networks. ArXiv, abs/2308. 10436, 2023.
Jumper. J., Evans. R.. Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool. K., Bates, R., Zidek, A., Potapenko, A., et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873)5583-588, 2021.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Kipf. T., Fetaya, E., Wang. K.-C,, Welling, M.. and Zemel, R. Neural relational inference for interacting systems. In Dy, J. and Krause, A. (eds.). Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. htt.reeeddnnn.......errees/eeeee
PMLR, 10-15 Jul 2018. URL WtLLhttps://proceedniie mlr.
html
Kondor, R. and Trivedi, S. On the generalization of equivariance and convolution in neural networks to the action of compact groups In ICML, 2018.
Kondor, R., Lin, Z., and Trivedi, S. Clebsch Clebsch-gordan nets: a fully fourier space spherical convolutional neural network. Advances in Neural Information Processing Systems. 31, 2018.
Kufel, D., Kemp, J., and Yao, N. Y. Approximatell neural networks for quantum many-body physics. 2023, URL htttps://app semant D :0: 268031561
Liao, Y.-L. and Smidt, T. Equiformer: Equivariant graph attention transformer for 3d atomistic graphs.
In International Conference on Learning Representations, 2023. URL htttps://oppereelltt
net/forumiid-Kie KwmPf ARgOTD.
Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL http::////pppnereeeeeeeeeee////s hett/fforumm1dddkk6.......
Luo, S., Chen, T., and Krishnapriyan, A. S. Enabling efficient equivaaraat operations in the fourier basis via gaunt tensor products. arXiv preprint arXiv:2400000000022666. 2024.
Maile, K., Wilson, D. G., and Forré, Equivariance-aware architectural optimization of neural networks, 2023.
Maron, H., Ben-Hamu, H., Shamir, N., and Lipman, Y. Invariant and equivariant graph networks. ArXiv, abs/1812 812,09902, 2019.
Mondal, A. K., Panigrahi, S. S., Kaba, S.-O., Rajeswar, S., and Ravanbakhsh, S. Equivariant adaptation of large pretrained models. In Thirry-eevntth Conference Neural Information Processing Systems, 2023. U httphtttt//eeeeeiiieee http://aaarrriieeeeeeooeoorrrro (it-:rrnmmmnd-mmmm000.00222
Ocampo, J.. Price. M. A., and McEwen, J. Scalable and equivariant spherical CNNs by discrete- continuous (DISCO) convolutions. In The Eleventh International Conference on Learning Repre- sentations, 2023. URL
Ordoñez-Apraez, D., Turrisi, G., Kostic, V., Martin, M., Agudo, A., Moreno-Noguer, F., Pon- til, M., Semini, C., and Mastalli, C. Morphological symmetries in robotics. arXiv preprint ar55522402.1555,, 2024.
12

---

Otto, S. E., Zolman, N., Kutz, J. N., and Brunton, S. L. A unified framework to enforce, discover. and promote symmetry in machine learning, 2024. URL
Passaro, S. and Zitnick, C. L. Reducing so (3) convolutions to so (2) for efficient equivariant gnns. In International Conference Machine Learning, pp. 27420-27438, PMLR, 2023.
Pearce-Crump, E. Brauers group equivariant neural networks. In International Conference on Machine Learning, pp. 27461-27482. PMLR, 2023.
Petrache, M. and Trivedi, S. axproximatioon-eeeraliinnn
trade-offs under (approximate) group cquivariance. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 31936-61953 Curran As- sociates, Inc., 2023. URL http::t:
neurips cc/paper___ files/ httpp//aaaper/////////// pdf
Petrache, M. and Trivedi, S. Position paper: Generalized grammar rules and structure based generalization beyond classical equivariance for lexical tasks and transduction. arXiv preprint arXiv:2402.01622 2024.
Pollard, D. Empirical processes: Theory and applications. In NSF-CBMS Regional Conference Series in Probability and Statistics, pp. i-86. JSTOR, 1990.
Qi, C. R., Su, H., Mo, K., and Guibas, L. J. Pointnet: Deep learning on point sets for 3d classification and segmentation. 2016. URL http:://atatti wwwg/og/////eeeeoooo http//bb/1612.... 00593 cite arxiv: 1a1i:::612.005933
Ravanbakhsh, S., Schneider, J. G., and Póczos, B. Equivariance through parameter-sharing. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference Machine Learning, JCML 2017, 7, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 2892 2901. PMLR, 2017. URL http://proccedinng mlr html.
Romero, D. W. and Lohit, S. Learning partial equivariances from data. Advances in Neural Information Processing Systems, 35:36466-36478, 2022.
Satorras, V. G., Hoogeboom, E., Fuchs, F., Posner, I., and Welling, M. E(n) equivariant normalizing flows for molecule generation in 3d. ArXiv, abs/2105.09016 2021.
Townshend, R. J. L., Eismann, S., Watkins, A. M., Rangan, R., Karclina, M., Das, R., and Dror, O. Geometric deep learning of rna structure. Science, 373:1047 2021.
van der Ouderaa, F. A., Romero, D. W., and van der Wilk, M. Relaxing equivariance constraints with non-stationary continuous filters. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, Dec 2022. doi: 444550//RXXI.2204.00777 org ab2///04./222....
URL https:///////
Villar, S., Hogg, D. W., Storey-Fisher, K., Yao, W., and Blum-Smith, B. Scalars universal: Equivariant machine learning, structured like classical physics. Advances in Neural Information Processing Systems, 34:28848-28863, 2021.
Wang, D., Walters, R., Zhu, X., and Platt, R. Equivariant $4$ learning in spatial action spaces. In 5th Annual Conference on Robot Learning, 2021. URL htttp:://ppenneriee..tt////o IScz42A3ICI
Wang, R., Walters, R., and Yu, R. Approximately equivariant networks for imperfectly symmetric dynamics. In International Conference on Machine Learning. PMLR, 2022.
Wang, R., Walters, R., and Smidt, T. Relaxed octahedral group convolution for learning symmetry breaking in 3d physical systems. In NeurlPS 2023 Al for Science Workshop, 2023. URL httpps //openreview wwwttfeuet/iiiiie ott.:///
Wang, R., Hofgard, E., Gao, H., Walters, R., and Smidt, T. E. Discovering symmetry breaking in physical systems with relaxed group convolution, 2024. URI.
022298
13

---

Wang, Y., Sun, Y., Liu, Z., Sarma, S. E., Bronstein, M. M., and Solomon, J. M. Dynamic graph emn for learning on point clouds. ACM Transactions on Graphics (TOG), 2019.
Weiler, M. and Cesa, G. General B(2)-Equivariant Steerable CNNs. In Conference on Neural Information Processing Systems (NeurIPS), 2019.
Winkels, M. and Cohen, T. Pulmonary nodule detection in scans with equivariant enns. Medical image analysis, 53:15-20 2019.
Xu, Y., Lei, J., Dobriban, E., and Daniilidis, K. Unified fourier-based kernel and nonlinearity design for equivariant networks on homogeneous spaces. In Chaudhuri, K., Jegelka, S., Song. L.. Szepesvári, C., Niu, G., and Sabato, S. (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 1459444461.... PMLR, 2022. URL http:tt////rccccdnnggg mlr.press/ v162/xu22e html.
Zhu, X., Wang, D., Biza, O., Su, G., Walters, R., and Platt, R. Sample efficient grasp learning using equivariant models. CoRR, abs/2202.09468, 2022. Utt. ortp/b:/2222222222222 09468.
14

---

A Appendix/ Supplemental Material
A.1 Training Details
In this section, we provide additional details for the application of our framework in the experiments presented in this work. We fix the weight of the regularization term to be Xreg 0.01 for all the experiments. We arrive that the above value for the hyperparameter Xveg by performing grid-search using cross validation, as described in more detail in Section A.2 Additionally, except the corresponding ablation study, we use the scheduling of the value of as described Section The variance reported is over random trials of the same experiment with different seeds. We run all the experiments on NVIDIA A40 GPUs. For the model-specific training details:
Point Cloud Classification: We use as baselines the VN-PointNet and VN-DGCNN network architectures described in the work Deng al. (021). For the relaxed version of VN- PoitNet we train for 250 epochs using the Adam optimizer (Kingma & Ba 2015), with an initial learning rate of 10 -3 that we decrease every 20 epochs by a factor of 0.7, and weight
decay equal to 10m4 For the relaxed version of VN-DGCNN we train for 250 epochs using stochastic gradient descent, with an initial learning rate of 10-3 that we decrease using cosine annealing, and weight decay equal to 10-1 The batch size used was 32.
Nbody partiele simulation: We train the relaxed version of SEGNN (Brandstetter et al. 2021) for 1000 epochs using Adam optimizer (Kingma & Ba 2015) with learning rate of 5 * 10-4, a weight decay of 10-12 and batch size of TOO. We report the test MAE for the model at the training epoch that achieved the minimum validation error.
Molecular Dynamics Simulation: We train the relaxed version of Equiformer (Liao & Smidi 2023) for 1500 epochs using AdamW optimizer (Loshchilov & Hutter, (2019) with an initial learning rate of 10 5 that we decrease using cosine annealing, and with weight decay equal to 10-6, The batch size used was 8. We use the network variant with max representation type set to Limax 2
Approximately Equivariant Steerable Convolution: We train the approximately equiv- ariant steerable convolution proposed in Wang et al." (2022) after we apply our additional relaxation. We modify the same architecture used in the original work which contains layers of approximate equivariant steerable convolutions. We train for 1000 epochs using the Adam optimizer (Kingma & Ba 2015). We use an initial learning rate of 10-4, that we decrease at each epoch by 0.95. We perform early stopping, where the stopping criterion is that the mean validation score of the last epochs exceeds the mean validation score of the previous 5 epochs.
A.2 Choice of Hyperparameters
In all the experiments, apart from the weight Xreg of the proposed regularization term, the same hyperparameters as the ones used by the baseline methods we compare with. For the choice of Aveg we perform hyperparameter grid search using cross-vaie validaaiio with an 80%-20% split of the original training set of ModelNet40 into training and validation. of VN-Pointnet model trained with our method on the 80% training split and evaluated on the 20% validation split for different values of Areg- We observed that the best value of Xreg is relatively robust across tasks, we performed an extensive hyperparameter search for the task of point cloud classification, and we used the found value Areg 0.01 across all other tasks.
the performance
A.3 Computational and Memory Overhead of proposed method
The computational overhead introduced by our method only affects the training process. During inference, after the projection to the equivariant space, the retrieved model has the same architecture as the corresponding base model to which we apply our method on, thus it also has the same computational and memory requirements. In Table we show the cost of our method, in terms of training time and number of additional parameters. While our proposed method introduces additional parameters during training, due to the parallel nature of the unconstrained non-equivariant term, the overhead in training time can be limited given enough memory resources. As result, while the additional parameters are approximately three times the parameters of the base model the increase in the training time is below 10% of the base training time.
15

---

ModelNet Classification Validation Accuracy vs Ang
Accuracy (%)
0.001
0.002
0,005
0,000
8020
0050
0100
Ang
Figure 5: ModelNet40 classification accuracy on the validation set using our proposed method with different values of Xreg- The base model used was the VN-PointNet. The model trained split of the training set containing 80% of the training data. The other 20% of the data were held out as the validation set used to evaluate the model.
Table Additional Number of parameters and Computational Overhead introduced by the proposed method
Number of Parameters
Additional Parameters
Time per Epoch (Base Model)
Time per Epoch (Ours)
Model
(Base Model)
(Ours)
VN-PoiniNet
1.9M
6.4M
75s
80s
VN-DGCNN
1.8
6.2M
148s
154s
Equilormer
3.4M
10M
52s
57s
A.4 Ablation on 0 Scheduling and Equivariant projection
During the later stages of training our proposed @ scheduling slowly decreases the level of relaxation of the equivariant constraint, bringing the model closer the equivariant space. This process allows the model to smoothly transition from the relaxed equivariant case to the exact equivariant one. In Figure/o we show a comparison of the performance of model trained with our proposed 0 scheduling and model trained with a constant 0 before and after it is projected to the equivariant space, While the performance of the relaxed equivariant model with constant is close to the performance achieved by our method, we can observe a sudden drop in performance once it is projected back to the equivariant space. On the other hand, our proposed scheduling of allows the model to return the equivariant space by the end of training without showcasing such significant performance drop.
Classification VN-PointNet
Test Accuracy
0.5
Ours (Fuil Method)
0.2
Constant Theta (after projection) Constant Theta (betore projection)
25c
200
Epoch
Figure 6: Comparison of the performance on ModelNet40 classification (300 points), for model trained with our method and model trained with constant value of 0, for which the level of cquivariant error is controlled only by the regularization term. For the latter method we show results both before and after the projection to the equivariant space.
16

---

A.5 Additional comparison with Methods using Equivariant Adaptation/Fine Tuning
As described in Section while our work focuses on improving the optimization of equivariant net- works, the works of Mondal et al. (2023) (Equiv-Adapt) and Basu et al. (2023b) (Equi-Tuning) focus on circumventing the need of optimizing equivariant network by performing equivariant adaptation or fine tunining of non-equivariant models. Since such methods have different focus, mainly utilizing already pre-trained non-equivariant models to solve equivariant tasks, straightforward comparison can be challenging. Nevertheless, in T we provide comparison with our proposed method on the task of sparse point cloud classification. We can see that our method outperforms Equiv-Adapt and has performance close to the one achieved by Equi-Tuning which requires multiple forward passes during inference.
Table Comparison of our proposed method with previous works performing equivariant adaption or finetuning, on ModelNet40 classification (Base model: VN-PointNet), Here it is important to note that the case of Equi-Tuning, equivariance is achieved by group averaging. As a result, during inference the model is required to perform multiple forward passes, which slows down the method's inference.
Equiv-Adapt
Equi-Tuning Tuning
Original VNN
Ours
66.3%
74.9%
66.4%
74.5%
A.6 Limitations
As describe in Section 3 work focuses the assumption that the equivariant NN satisfies the equivariant constraint by imposing it in all its intermediate layers. Although this assumption is general enough cover a large range of state-of-the-ard equívariant architecture, it doesn't apply to all possible equivariant networks since it is possible to ensure overall constraint satisfaction using different methodology. Additionally the proposed regularizers in Section 3.J can be applied to tasks where the symmetry group is matrix Lie group or a discrete finite group. Extending our proposed framework to arbitrary symmetry groups is future research question that addressed this paper.
A.7 Broader Impact
This paper focuses the question of improving the optimization of equivariant neural networks. Such equivariant networks are currently used to solve tasks different fields- ranging from computer vision to computational chemistry. As result, its broader societal impact is highly dependent on the specific network it enables optimizing.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims and contributions of the paper are included at the end of the introduction in Section [ Similarly, the abstract of the paper reflects these claims
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of this work in Appendix A.6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairnsss.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
The paper doesn't 't provide any theoretical result in the form of a proof
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: In the experimental Section f and in the supplementary material in Section A.1 we disclose all the information needed to reproduce our results by citing the appropriate previous works. Additionally, we provide the set of hyperparameters and training details that are novel for our work. Finally in Section 3.4 4 we provide description of how our framework can be implemented for the set of network architectures that we use in the experiments. Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
19

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: In the abstract, provide link to GitHub repository containing all the code and instructions necessary to implement our proposed optimization method. Additionally, all of the data used in the experiments publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release code and data, we understand that this might not be possible, so No is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
Coaee//dddeSSmmmemeee issiiisssionn osiicc) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the required configurations for the different experiments are provided in the Section and Section A.1 1.1) of the appendix
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide the variance of our experimental results to random initializa- tion for all the experiments except the molecular dynamic simulation due its high computational cost.
Guidelines:
The answer NA means that the paper does not include experiments.
20

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Section A. of the appendix we disclose the computational resources used for the experiments in this work.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http////hhhi////////////o//////p
Answer: [Yes]
Justification: Our research follows the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the broader impact of our work in Section A.7
21

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
We do not plan to release any data or pre-trained models
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: This paper uses only publicly available data and code for which it credits their creators appropriately.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
22

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, datasets.paperssiii pwperspi thcode www.coo///aaaaa com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper doesn't 't release any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper doesn't 't include crowdsourcing or any research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper doesn't 't include crowdsourcing or any research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
23

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "GC46ZRFUdY", "reviewer": "Reviewer_RSZ3", "summary": "Starting from the consideration that equivariant neural networks, though effective for tasks with known data symmetries, are hard to optimize and require careful hyperparameter tuning, this study proposes a framework to improve their optimization process. This is done by temporarily relaxing the equivariance constraint during training. By adding and progressively reducing a non-equivariance term in intermediate layers, the model explores a broader hypothesis space before converging to an equivariant solution.", "questions": "1) What are the additional computational/memory costs of the proposed method? Is  there a trade-off between the achieved increased optimization stability and  exploiting the proposed procedure?\n\n2) Could the authors give some examples where the proposed method is not applicable, i.e. the  symmetry group is not a matrix Lie group or a discrete finite group? And do they believe that such settings are common or not, i.e. is the proposed method general enough to be applied to real-world scenarios?", "limitations": "The authors adequately addressed the limitations.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is clearly written, the topic is significant and of interest to the community.", "weaknesses": "Experimental results, despite confirming the theoretical considerations, seem to  not compare with competitor methods (the ones reporte in Related works section, e.g. Mondal et al. (2023),  Basu et al. (2023b) etc.) or to describe and analyze the additional computational costs (time/memory) of the provided optimization procedure (also in this case, compared to such alternative approaches).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive assessment of our paper and for the comments. Below we address all of the points the reviewer has raised.\n\n\n## Comparison with works on Equivariant Adaptation/Fine-tuning of pre-trained models\nThank you for raising this point. While it is true that both our method and the works of of Mondal et al. (2023), Basu et al. (2023b) have a similar motivation i.e. that the optimization of equivariant models can be more challenging than their non-equivariant counterparts, our paper addresses a different research question from their work. Specifically, our paper focuses on improving the optimization of equivariant architectures themselves, while the works of Mondal et al. (2023) and Basu et al. (2023b) focus on techniques that bypass the need for training equivariant models. Specifically, they focus on creating equivariant functions by using pre-trained non-equivariant models e.g. by canonicalization (as in Mondal et al.). \n\nWe believe that the difference in focus between our work and that of Mondal et al. (2023) and Basu et al. (2023b), does not quite permit a straightforward and fair comparison. Nevertheless, we include a comparison on a task of point cloud classification (ModelNet40) below. \n\n| Method | PointNet | DGCNN|\n|----------|----------|----------|\n| Mondal et al. (2023)   |  66.3\\%   | 90.4\\%   |\n| Basu et al. (2023b)   | 74.9\\%  | 89.1\\%   |\n| Original VNN   | 66.4\\%  | 88.5\\% |\n| Ours  | 74.5\\% | 92.0\\%  |\n\nHere it is important to note that in the case of Basu et al. (2023b), equivariance is achieved by averaging the results over multiple transformed inputs. As a result during inference, the model is required to perform multiple forward passes, which slows down the method's inference time.\n\nWe will be happy to include this comparison in the final version (in the main paper or the appendix) if the reviewer believes that it will benefit the overall narrative of the paper.\n\n## Additional Details on Computational and Memory costs of the proposed method\nWe provide a table of the computational overhead of our method compared to the baseline models. Additionally, to illustrate the overhead on the memory constraint we provide a comparison on the number of learnable parameters between our method and the baseline equivariant model. We would like to note that this overhead is only during the training of the models since during inference we remove the additional relaxation layers.\n\n| Model Type | Number of Parameters    (Base Model) | Additional Parameters         (Ours)  | Time per Epoch (Base Model) | Time per Epoch    (Ours) |\n|------------|--------------------------------------|---------------------------------------|-----------------------------|--------------------------|\n| PointNet   | 1.9M                                 | 6.4M                                  | 75s                         | 80s                      |\n| DGCNN      | 1.8M                                 | 6.2M                                  | 148s                        | 154s                     |\n| Equiformer | 3.4M                                 | 10M                                   | 52s                         | 57s                      |\nWe will add these details in the appendix. \n\n## Trade-off for optimization stability \n\nCould the reviewer clarify further? We might be misunderstanding and would be happy to discuss. In general, however, we think that the optimization stability of our method depends on the projection error term which can affect a tradeoff. For perfectly equivariant models in which we don't do any relaxation, we would expect them to be somewhat harder to optimize than a setup with relaxed weights that don't have high projection error. In cases where the projection error is too high, the quality of optimization will go down. Note that in our proposed method we can control the projection error by annealing the value of the $\\theta$ parameter of equation 2.\n\n## Examples where the proposed method is not applicable\n\nThe approach will generally work for a variety of groups of real-world interest. The approach as described will work for compact Lie groups such as $S(1), SO(n), O(n)$, and their finite subgroups like $Z/NZ$, as well as their quotients and products. The approach can be made to work for certain non-compact Lie groups which are reductive for which we can write expressions for the bracket, such as the Lorentz group. It might be challenging for the approach to work for permutation groups.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JRkDISU7fs", "reviewer": "Reviewer_3CAa", "summary": "The work proposes a method for improving generalization by relaxing hard equivariance and minimizing the equivariance error as additional regularizer during training.", "questions": "1. Projecting back to equivariance models during testing. The first contribution mentions “projecting back to the space of equivariance models during testing”. where is this method? To my understanding, the model is not actually projected, but the error in the projection is rather minimized (but not zero). I could have misunderstood this aspect?\n2. Regularization term for deep neural networks. In line 165, a regularization term is proposed to encourage equivariance solutions. It seems this term is only for a single layer. For multiple layers, how are the relative importances between layers chosen? Uniform? Also, it is not clear to me how the overall regularization strength \\lambda_reg should be chosen. Cross-validation? Similarly for \\theta.", "limitations": "The main contribution of the paper seems to be adding a regularization term that penalizes the ‘equivariance error’. Several works have proposed similar losses and minimizing such regularizers in the training objective. The paper lacks comparison or discussion between different choices of such regularizers, yet alone an empirical comparison. To me it is not clear what the main contribution of this paper is?", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "Symmetries play an important role in machine learning and deep learning specifically. There has been recent attention to relaxed forms of equivariance, making it a relevant topic. The paper is well-written.", "weaknesses": "There is a lack of attribution to related work, which results in a false sense of novelty. Although the paper does cite many papers on (relaxed) equivariance, it does not always give proper attribution to their contributions. Especially, since several cited papers in the related work section have proposed forms of relaxed equivariance and even minimizing such “relaxation errors” as regularization objective - either in their main method or as a baseline. Yet, the paper claims that “ solutions don’t directly focus on the optimization process itself”. This gives a false impression that regularizing the amount of equivariance in the loss is novel, while it is not. As such, to me it is not entirely clear what the contribution of this paper is.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for sharing constructive feedback. We agree that we can improve attribution to related work to emphasize the differences from our work. It is certainly true that there are many works on relaxed equivariance, some of which also employ regularization terms that penalize relaxation errors. However, these works tend to assume that the data itself has some (possibly) relaxed symmetry. The regularization term can then be seen as trying to match the symmetry encoded in the model to the symmetry of the data itself. In our case, the main difference---although it may seem subtle---is that we aren't looking to correct for model misspecification, although we include experiments for approximately equivariant NNs too. We try to make the case that even if we assume that the model is correctly specified, relaxing the equivariance constraint during optimization and then projecting back to the equivariant space can itself help in improving performance. Within the equivariant NNs community it is a common observation that equivariant NNs are harder to optimize than their non-equivariant counterparts [1][2][3]. We take a step towards examining this question in more detail as we believe that this particular hasn't seen exploration specifically. \n\nWith the above context in mind, we expand on each of your points below.\n\n## Paper Contribution compared to related work\nAs we said, while there are multiple works on different forms of relaxed equivariance, that also introduce regularization by minimizing the equivariance error, the main difference with our work is that they always remain in the space of relaxed equivariance models. Furthermore, the regularization solves for model mis-specification. We believe that the contribution of our work lies in the projection of the models back to the equivariant space which, contrary to previous works, guarantees that the solution has zero (or fixed) equivariant error. Thus, we are not aiming to address model mis-specification. However, we will improve the attribution to existing work to emphasize this difference, and to highlight the observation that optimizing equivariant networks can be harder than their non-equivariant counterparts.\n\n\n## Projecting back to equivariance models during test\nThank you for your comment. We believe there might be a misunderstanding regarding the projection back to the equivariant space during testing. In Section 3 we define the approximate equivariant linear layer as:\n$$f(x)=f\\_e(x)+\\theta Wx$$\n\nwhere $f\\_e(x)$ is an equivariant linear layer and $\\theta Wx$ is an unconstrained term. As we mentioned in lines 142-143, we can project this linear layer to be exactly equivariant by setting $\\theta=0$. In that case, only the equivariant part of the layer is activated $f(x)=f\\_e(x)$ which guarantees that the layer and as a result the overall model is equivariant and has exactly zero equivariant error. During inference, we set $\\theta=0$, resulting in an exact equivariant model. To improve clarity, in addition to the sentence in lines 142-143 we added an explicit note that the operation of setting $\\theta=0$ is what we refer to as projection in the rest of the paper.\n\n## Regularization Term for Multiple Layers\nAs we show in the overall training objective (Section 3.3, line 189) the regularization terms, derived for each individual layer, are added in the loss function with the same weight $\\lambda\\_{reg}$. \nWhile it is possible to introduce different weights for the different layers, these weights will heavily depend on the specific network architecture and will increase the complexity of the method. In order to keep the method simple and easily applicable to different architectures we choose to simplify the solution and use the same weight for all layers.\n\n## Choice of hyperparameters\nFor the choice of the hyperparameter $\\lambda\\_{reg}$ we performed a typical grid search using cross validation with a 80\\%-20\\% split of the training set into training and validation set. We found this value to be relatively robust across tasks, so we performed it on the task of point cloud classification and used the value in all other tasks. We include a figure (Figure 2) in the attached rebuttal PDF which shows how the value of $\\lambda\\_{reg}$ affects the performance of the method while using VN-PointNet as the baseline model. We will add these details and the additional figure in the Appendix of the final version of the paper. We can also include results tuning them individually in the appendix if the reviewer thinks it will be helpful. However, we must note that the accuracy shown in the figure is for the validation set when the model is trained on the 80\\% of the training set. The results shown in the paper are when we train the model on the complete training set (after we have chosen the hyperparameter $\\lambda\\_{reg}$).\n\nRegarding the parameter $\\theta$: since we perform the scheduling as described in section 3.2, we are not required to do hyperparameter search. The main constraint for the choice of $\\theta$ is that it needs to arrive at zero at the end of training. As shown in the ablation in section 4.1 if we do not perform $\\theta$ annealing we observe deterioration in performance. \n\n\n**References used in this response**\n\n[1] Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs\nYi-Lun Liao, Tess Smidt, arXiv:2206.11990\n\n\n[2] Discovering Symmetry Breaking in Physical Systems with Relaxed Group Convolution\nRui Wang, Elyssa Hofgard, Han Gao, Robin Walters, Tess E. Smidt, arXiv:2310.02299\n\n[3] Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Network\nRisi Kondor, Zhen Lin, Shubhendu Trivedi, arXiv:1806.09231", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WUpJBSNj2q", "reviewer": "Reviewer_GoLr", "summary": "The paper proposes relaxing the equivariance constraint on an equivariant network during training. This is done by adding free weights to equivariant linear layers but setting the free weights to zero after training. Further, two regularizations are introduced to stabilize the training: a Lie derivative term encouraging the free weights to be close to equivariant and a term encouraging the influence of the free weights to be low compared to the equivariant weights. The approach is evaluated on several equivariant tasks, showing improved performance compared to the baseline of non-relaxed optimization.", "questions": "1. The proposed parameterization of equivariant layers during training is $f(x) = f_e(x) + \\theta W x$ where $f_e$ is equivariant. $W$ is also encouraged to be close to equivariant through Lie derivative regularization. Would it be possible to remove $f_e$ and $\\theta$ to parameterize $f(x)=W x$ as a non-equivariant layer that is regularized by the Lie derivative loss? The projection to the equivariant subspace at the end could be group averaging: $W\\mapsto \\int_{g\\in G} \\rho(g^{-1}) W \\rho(g) \\mathrm{d}g$. Is there something that speaks against such an approach?\n2. What is the introduced overhead during training? In terms of time and memory costs.\n3. What is the performance of the obtained trained network without projecting to the equivariant subnetwork? I.e. is the approximately equivariant network better than the equivariant?\n\nMinor:\n- Line 243 \"table 2\" -> \"Figure 2\"", "limitations": "-", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper provides a solid contribution to the understudied topic of optimizing equivariant networks. As far as I am aware, the idea has not been studied in the literature before.\n2. The presented experiments show a small but consistent benefit using the proposed approach.", "weaknesses": "1. There is no theoretical motivation for why the proposed approach should work.\n2. The optimization will be heavier using the proposed approach since a large number of additional parameters are introduced. See also Question 2 below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive assessment of our work. For the questions you raise, we attempt to address them below. Please let us know if we can provide further clarifications. \n\n## Theoretical motivation\nWithin the equivariant NNs community, especially amongst practitioners using such networks in scientific applications, it seems to be a fairly common observation that such networks can be harder to optimize as compared to their non-equivariant counterparts [1][2][3]. We wanted to take a step towards exploring this question and its attendant space, which we believe to be a good enough contribution to the community. We have started to form some theoretical justifications for how/why such approaches should work or be modified. We hope to explicate on a theory for future work. Generally, working out an optimization-generalization trade-off is hard (say compared to working a tradeoff between generalization and approximation). Here we would like to work it out for equivariant networks versus non-equivariant ones.  \n\n## Regarding the additional Optimization Cost\nThank you for the question. As you correctly point out the cost of optimization of our method will be higher than a baseline equivariant network due to the additional parameters. Nevertheless, due to the parallel nature of the added unconstrained component, the main overhead of the method is on the additional memory required to store the added parameters. This number of additional parameters makes the model of similar size to the typical unconstrained non-equivariant models, which is expected since during training we optimize over a space that is larger to the constrained space of equivariant models. We would like to note that contrary to other methods on approximate equivariant models, that require to access additional parameters both in training and inference, our method requires additional computational resources only during training. As a result, after training is completed our proposed method will not have an effect on inference time.\nWe provide a comparison between the time cost of our proposed training process and the baseline training of different equivariant models. Also to showcase the memory overhead we provide a comparison between the number of learnable parameters between a model trained with our method and an equivariant model trained with a standard training process:\n\n| Model Type | Number of Parameters    (Base Model) | Additional Parameters         (Ours)  | Time per Epoch (Base Model) | Time per Epoch    (Ours) |\n|------------|--------------------------------------|---------------------------------------|-----------------------------|--------------------------|\n| PointNet   | 1.9M                                 | 6.4M                                  | 75s                         | 80s                      |\n| DGCNN      | 1.8M                                 | 6.2M                                  | 148s                        | 154s                     |\n| Equiformer | 3.4M                                 | 10M                                   | 52s                         | 57s                      |\n\nWe will add a discussion about the optimization cost of our method in the Appendix of the final version of the paper.\n\n## Regarding Suggestion in Question 1\n\nThanks for the interesting question. An important part of our proposed method is the ability to control the level of the equivariant relaxation explicitly, in addition to the implicit control that comes as a result of the regularization term. With our current parametrization, we can control the level of relaxation by controlling the value of  $\\theta$. The importance of this control and of the annealing of theta can be seen in Figure 2 (of the submitted paper), where in the case where only the Lie derivative regularization is applied with the value of $\\theta$ being constant, the performance of the method decreases.\n\nWe think that with the group averaging approach, it could be harder to have explicit control on the level of relaxation of the equivariant constraint during training. As a result when projecting back to the equivariant case it is possible that the performance gap between the relaxed and projected model might be large, even with the lie derivative regularizer.\n\nAdditionally, the operation of group averaging can be performed up to approximation in the case of continuous groups, since we need to sample the group elements to approximate the convolution integral.  \n\n## Performance of the Network without the Projection\nIn Figure 3 in the rebuttal PDF we added a comparison between the performance of a model trained with our method and the performance of a relaxed equivariant model before and after we project it to the equivariant space. For the relaxed model that we compare with, the $\\theta$ parameter is kept constant and the equivariant error is controlled only by the regularizer. We can observe that our method outperforms the relaxed equivariant model with constant $\\theta$ even before we project the latter in the equivariant space.\n\n## Typos\nThank you for pointing this out. We will correct it in the paper. \n\n**References used in this response**\n\n[1] Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs\nYi-Lun Liao, Tess Smidt, arXiv:2206.11990\n\n\n[2] Discovering Symmetry Breaking in Physical Systems with Relaxed Group Convolution\nRui Wang, Elyssa Hofgard, Han Gao, Robin Walters, Tess E. Smidt, arXiv:2310.02299\n\n[3] Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Network\nRisi Kondor, Zhen Lin, Shubhendu Trivedi, arXiv:1806.09231", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zy8o3SxEsn", "reviewer": "Reviewer_b5uM", "summary": "The paper proposes to relax the equivariance constraints in equivariance networks by adding a non-equivariant residual in the network weights. The methodology can be applied to different equivariant architectures, e.g. vector neurons, SE(3)-equivariant GNNs, Equiformers. Besides these strictly equivariant networks, the proposed framework can also extend to the approximately equivariant networks by modulating the strength of the unconstrained network weights. Experiments suggest moderate performance increases on a variety of tasks and equivariant architectures when applied the proposed framework.", "questions": "1. The scheduling choice of $\\theta$ in Section 3.2 seems arbitrary. Have you tried other scheduling choices, e.g. initializing with zero and constantly increasing it? What is the intuition for the current choice?\n\n2. How sensitive is the model to the regularization coefficient $\\lambda_\\text{reg}$?\n\n3. L193: \"During inference, we evaluate only on the equivariant part of the model.\" It would be interesting to also see the result for the full model (i.e. including W). I wonder how equivariant and how large W would be under the current regularization. I'm asking because approximately equivariant networks have proved to outperform strictly equivariant networks on certain tasks with imperfect symmetries (Wang et al 2022). As you have both the non-equivariant network and the equivariant one, I'm curious about the comparison between them.\n\n4. Is it possible to have different weighing coefficients for each network layer in the Lie derivative regularization and projection error regularization? As these errors can accumulate after passing through multiple layers, I think it is intuitively reasonable to have larger weights for the first few layers.\n\n### References\n* Wang, Rui, Robin Walters, and Rose Yu. \"Approximately equivariant networks for imperfectly symmetric dynamics.\" International Conference on Machine Learning. PMLR, 2022.\n* Wang, Rui, Robin Walters, and Tess E. Smidt. \"Relaxed Octahedral Group Convolution for Learning Symmetry Breaking in 3D Physical Systems.\" arXiv preprint arXiv:2310.02299 (2023).\n* Otto, Samuel E., et al. \"A unified framework to enforce, discover, and promote symmetry in machine learning.\" arXiv preprint arXiv:2311.00212 (2023).", "limitations": "The authors have discussed the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper addresses an important problem: overcoming the optimization challenges for equivariant neural networks. The methodology is clearly motivated and easy to follow. The proposed framework of adding an unconstrained component in the network weights seems general enough to be implemented on various (though not all) equivariant architectures, which the authors have demonstrated with quite a few examples.", "weaknesses": "* Formatting: the paper is visually difficult to read because of the incorrect use of the parentheses in citations.\n\n* Theoretical contribution: as far as I understand, the motivation for relaxing the equivariance constraints is (i) the equivariant networks can be difficult to optimize, or (ii) the data have imperfect symmetry. The paper does not include any theoretical evidence of how the proposed framework of adding unconstrained weights can be helpful in these scenarios.\n    * Specifically, for (i), the paper did not point out what could be the specific challenges during the optimization of equivariant networks, compared to unconstrained optimization.\n    * And for (ii), previous works have shown the error bound of approximately equivariant networks on imperfectly symmetric data (Wang et al 2022) and proposed how to find the symmetry-breaking factors (Wang et al 2023). Compared to these works, I feel this paper didn't provide enough analysis e.g. of equivariance error of the proposed network, or how to interpret the learning result and possibly identify the imperfect symmetry in data.\n\n    Also, many of the results in the paper are already well-known, e.g. the equation about the Lie derivative. Several works have already proposed to use the Lie derivative as a regularization to encourage equivariance, e.g. Otto et al 2023.\n\n* Significance of experimental results: in the experiments, the proposed method only increases performance by a little. Also, in Figure 2, it seems that some models have not converged after 200 epochs. It would be better to train for more epochs and include the full results. In Table 1, the difference between Equiformer and your method is very small. It's hard to verify the significance of this result without looking at the error bar.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive comments as well as raising many points for clarification. We try to address the points raised below. \n\n## Citation Format\nWe apologize for the incorrect formatting of citations and appreciate the reviewer for pointing it out. We will correct this in the final version of the paper. \n\n## Theoretical contributions\n\nUnlike in other works on relaxed equivariance, here we specifically focus on the setting where the training data distribution and the model don't have a mis-match in terms of symmetry. As a result the performance improvements come from the fact that the proposed relaxation process can ease the optimization of equivariant networks. As we also noted with other reviewers, amongst equivariant NNs practitioners it is now a somewhat common observation that equivariant NNs can be harder to optimize than their non-equivariant counterparts [1][2][3]. However, the question itself remains unexplored. We take a step towards examining this question in more detail --- we believe that identifying processes that can provide empirical improvements in the optimization of such networks is a reasonable standalone contribution that can be valuable to the community. \n\nHowever, we have been working on developing theoretical models that develop an optimization-generalization trade-off which could perhaps provide some rationale for why equivariant models might be harder to optimize. We leave explicating on this for a future paper, since the situation can get quite involved.  \n\n## Significance of experimental results\nWe thank the reviewer for the feedback regarding the result shown in Figure 2 of the paper. We followed the suggestion and let the models train over the limit of 200 epochs and we show the results in Figure 1 of the attached rebuttal PDF. We see that indeed the additional epochs allow the model to converge, increasing their performance in some cases. It is important to note that while we extended the training epochs we kept the scheduling of theta the same, meaning that theta becomes equal to zero from epoch 200 and onwards. In the original submission we chose to follow the exact setup used from the baseline method (that used 200 epochs) to showcase how our method can provide improvements in performance without the need to tune again all of the training hyperparameters. However, we appreciate the comment of the reviewer and will include the updated figure in the final version of the paper. \n\nRegarding the error bars to quantify the significance of the results for the Equiformer experiment: due to the high computational cost of training a single equiformer model, there is a significant cost involved in providing detailed error bars. The original paper (Liao and Smidt 2023) did not include error bars in their reported results. However, we are actively training more models and we can provide the resulting error bars in the final version of the paper.\n\n## Choice of $\\theta$ scheduling.\nWe thank the reviewer for the comment. The main constraint for the scheduling of $\\theta$ is that we want to be equal to zero at the end of training so that the final model is equivariant. As a result, the choice of linearly increasing theta doesn't satisfy that constraint. Additionally, we observed that a warm up period in the beginning of training with a small $\\theta$ provides significant improvement in the training process. These two observations informed our choice of $\\theta$ scheduling where we linearly increase it for the first half of the training and then we linearly decrease it. After the suggestion of the reviewer we will add these observations to the Appendix of the paper. \n\n## Sensitivity to the regularization coefficient.\nIn Figure 2 in the attached rebuttal PDF we show the performance of the VN-Pointnet model for different values of the regularization parameter. This experiment was part of a hyperparameter search using cross-validaton on 80\\%-20\\% split of the original training set. As a result the documented performance represents models trained only on the 80\\% of the train dataset and evaluated on the other 20\\%.\n\n## Additional Details\n**Regarding evaluation on unprojected relaxed model:** \nWe appreciate the suggestion of ther reviewer regarding adding comparison between the relaxed non-equivariant model and the projected equivariant model. Figure 3 (included in the attached rebuttal PDF) shows a comparison between a model trained with our proposed method and a relaxed model before and after we project it to the equivariant space. For the latter model the $\\theta$ parameter is kept constant and the equivariant error is controled only through the regularization term. We can observe that, although the performance of the relaxed model before projection is close to the results achieved by our method, the control of the relaxation through our proposed  annealing of $\\theta$ benefits the performance. We will add this comparison in the final version of the paper.\n\n**Regarding the use of different weighting in the Lie derivative regularization:** It is true that a more general approach would use different regularization weights for each of the layers of the network. However the use of different weights per layer heavily depends on  the individual architectures and will introduce additional complexity to our method. Thus we chose to use the same weight for all the layers to make our method simpler and easier applicable to different tasks independently to the specific architectures used in the task.\n\n**References used in this response**\n\n[1] Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs\nYi-Lun Liao, Tess Smidt, arXiv:2206.11990\n\n\n[2] Discovering Symmetry Breaking in Physical Systems with Relaxed Group Convolution\nRui Wang, Elyssa Hofgard, Han Gao, Robin Walters, Tess E. Smidt, arXiv:2310.02299\n\n[3] Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Network\nRisi Kondor, Zhen Lin, Shubhendu Trivedi, arXiv:1806.09231", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all of the reviewers for their comments and constructive feedback on our paper. Here we would like to provide an overview of some of the main points of our individual responses to the reviewers and also take the opportunity to highlight the main contributions of our work:

 In this work, we propose a novel training procedure for equivariant neural networks that relaxes the equivariant constraint during training and projects back to the space of equivariant models during inference. As we discuss in more detail in our responses to reviewers 3CAa, b5uM, the differentiating factor between our paper and previous works on relaxed equivariance is that while we optimize over a larger space of relaxed equivariant models, at the end of optimization we project back to the space of equivariant models. This process allows us to learn models that have increased performance, compared to equivariant models trained with standard training, while at the same time, they have exactly zero equivariant error.
 
 Following the suggestions of the reviewers, in the attached rebuttal PDF  we provide additional ablations showing the effect of the individual component of our method on the overall performance (e.g. lie derivative regularization, $\theta$ scheduling, projection to the equivariance space). Specifically:
 
  -  Figure 1 extends the results of the ablation study provided in the submitted paper, where it shows the performance of different versions of our method when different components are removed.
  -  Figure 2 shows the sensitivity of our method to the choice of the regularization coefficient $\lambda_{reg}$.
  -  Finally, in Figure 3 we provide a comparison between a model trained with our proposed method and a relaxed equivariant model that is trained without the use of $\theta$ scheduling. In the case of the relaxed equivariant model with constant $\theta$, we show that its overall performance is worse than our method even before we project it in the equivariant space.

We believe that improving the training of equivariant neural networks is an important research question that can benefit the community and that our work is a positive step in that direction. We appreciate all the comments raised by the reviewers as they helped us enhance the presentation of this work, and we try to address all of them in our individual responses.

Meta-review: This paper advocates facilitating training of equivariant networks by relaxing the model’s equivariance during training. In particular, it adds and progressively removes the non-equivariant terms in the training process.

Reviewers found this paper clear and the motivation of improving training of equivariant models a good one. They considered the idea of progressing to equivariant solution during training mostly new, and appreciated the ``small yet consistent’’ improvement in results. Some concerns were raised regarding the similarity to previous works, where both the way the current paper adds non-equivariant terms, and the usage of the Lie derivative loss to encourage equivariance were used in previous works. However, the notion of forcing zero equivariance loss during training seems novel. Other concerns were the missing empirical and/or theoretical analysis, and the authors added during the rebuttal more experiments and some ablations of different components of their method. The authors also addressed issues of missing comparisons and memory+time cost tables. Overall, it seems most critical criticism of the reviewers were addressed. The only remaining negative reviewer is making relevant claims (mostly the contribution concerns mentioned above) but the overall impression is that the paper has merit and passes the bar for publication. 

We request the authors to incorporate the promised changes from the rebuttal in the camera ready version of this paper.
Decision: Accept (poster)