A General Theory of Correct, Incorrect, and Extrinsic Equivariance
Authors: Dian Wang
Xupeng Zhu
Jung Yeon Park
Mingxi Jia
Guanang Su
Robert Platt
Robin Walters
Venue: NeurIPS 2023 poster (2023)
Keywords: Equivariance
Deep Learning
Error Bound
Symmetry
TL;DR: We present a general theory when the symmetry of an equivariant network does not match the symmetry of the ground truth function.

Abstract: Although equivariant machine learning has proven effective at many tasks, success depends heavily on the assumption that the ground truth function is symmetric over the entire domain matching the symmetry in an equivariant neural network. A missing piece in the equivariant learning literature is the analysis of equivariant networks when symmetry exists only partially in the domain. In this work, we present a general theory for such a situation. We propose pointwise definitions of correct, incorrect, and extrinsic equivariance, which allow us to quantify continuously the degree of each type of equivariance a function displays. We then study the impact of various degrees of incorrect or extrinsic symmetry on model error. We prove error lower bounds for invariant or equivariant networks in classification or regression settings with partially incorrect symmetry. We also analyze the potentially harmful effects of extrinsic equivariance. Experiments validate these results in three different environments.

A General Theory of Correct, Incorrect, and Extrinsic Equivariance
Dian Wang
Xupeng Zhu'
Jung Yeon Park' Mingxi Jia
Guanang Su
Robert Platt Robin Walters
Northeestern University Brown University University of Minnesota @northeastern. Antttttt...........aaattttee. edu su0002650umn. edu
{uang. park. jungy, .r. pl-ppatt,,
walters) t
Abstract
Although equivariant machine learning has proven effective many tasks, success depends heavily on the assumption that the ground truth function is symmetric over the entire domain matching the symmetry in equivariant neural network. A missing piece in the equivariant learning literature is the analysis of equivariant networks when symmetry exists only partially the domain. In this work, we present a general theory for such a situation. We propose pointwise definitions of correct, incorrect, and extrinsic equivariance, which allow to quantify continu- ously the degree of each type of equivariance function displays. We then study the impact of various degrees of incorrect or extrinsic symmetry on model error. We prove eтror lower bounds for invariant equivariant networks in classification or regression settings with partially incorrect symmetry. We also analyze the poten- tially harmful effects of extrinsic equivariance. Experiments validate these results in three different environments.
1 Introduction
Equivariant neural networks [9, 10] have proven be an effective way to improve generalization and sample efficiency in many machine learning tasks. This is accomplished by encoding task-level sym- metry into the structure of the network architecture so that the model does not need to explicitly learn the symmetry from the data. However, encoding fixed type of symmetry like this can be limiting when the model symmetry does not exactly match the symmetry of the underlying function being modeled, i.e., when there is symmetry mismatch. For example, consider the digit image classifica- tion task. Is it helpful model this problem using model that is invariant to 180 180-degree rotation of the image? For some digits, the label is invariant (e.g., and 8). However, for other digits, the label changes under rotation (e.g., 6 and 9), suggesting that rotationally symmetric model would be inappropriate here. However, recent work [58] suggests that this not necessarily the case symmetric models are sometimes helpful even when symmetry mismatch exists between the problem and model. This raises the question do the advantages obtained by using a symmetric model outweigh the errors introduced by the symmetry mismatch?
f0x)
ybo)
(a) Correct
(b) Incorrect
(c)
Extrinsic
Figure 1: An example of correct, incorrect, and extrinsic equivariance. The ground truth func- tion f(x) shown in black and its probability density function p(z) is shown in orange. If we model f(x) using G-invariant network where G' is a reflection group that negates x, different f(x) and p(z) will lead to correct, incorrect, and extrinsic equivariance. See Section 3 for details.
* Work done as students Northeastern University
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

This paper makes four main contributions towards this problem. First, this paper extends the definitions for types of model symmetry with respect to true symmetry introduced in Wang et al. [58]. They classify models as having correct, incorrect, or extrinsic equivariance (see Figure 1), where correct means the ground truth function has the same symmetry as the equivariant model, incorrect means the ground truth function disagrees with the symmetry in the model, and extrinsic means the model symmetry transforms in-distribution data to out-of-distribution data. We generalize this system into a continuum of equivariance types to reflect the fact that a single task may have different proportions of correct, incorrect, and extrinsic symmetry across its domain. For example, in the digit classification task, 0 has correct equivariance, 6 has incorrect equivariance, and 4 has extrinsic equivariance.
Our second contribution is to introduce an analytical lower bound on model error in classification tasks resulting from incorrect model symmetry. This result can help guide model selection by quantifying error resulting from incorrect equívariance constraints. Our result generalizes that of Wang et al. [58] by removing the simplifying assumption that data density over the domain is group invariant. We prove the minimum error of an invariant classifier can be realized by assigning all data points in the same group orbit the label with the majority of the data density (Theorem 4.3).
Our third contribution is to develop new lower bounds on the L2 error for regression tasks in terms of the variance of the function to be modeled over the orbit of the symmetry group. Like our classification bound, this bound can assist in model selection in situations with symmetry mismatch.
Fourth, in contrast to Wang al. [58] who show benefits of extrinsic equivariance, we theoretically demonstrate its potential harm. We perform experiments documenting the error rate across the correct-extrinsic continuum. Finally, we perform empirical studies illustrating the ideas of the paper and showing that the lower bounds obtained in our analysis appear tight in practice. This suggests our analysis can assist practitioners select symmetry groups appropriate for given problem setting. Our code is available at https: // //////tttbbt//t wtt./ooonottteeeeeeeee pwwntw/ext.t....mptpppppppr.r..e htap//xt__hhheee eetttt//eetttttrr..tt
2 Related Work
Equivariant Learning. Originally used for exploiting symmetry in image domains [9, 10]. equivari-
ant learning has been very successful in various tasks including molecular dynamics [2, 4], particle
physics [6], fluid dynamics [59], trajectory prediction [53], pose estimation [31, 26, 32], shape com- pletion [7], robotics [48, 65, 21, 55, 49, 41, 23, 22, 45] and reinforcement learning [52, 54, 56, 39, 63). However, most prior work assumes that the symmetry of the ground truth function is perfectly known and matches the model symmetry. Wang et al. [58] go further and define correct, incorrect, and extrinsic equivariance to classify the relationship between model symmetry and domain symmetry. However, they do not discuss the possible combinations of the three categories, and limit their theory to compact group and invariant classification. Our work extends [58] and allows for a continuum of cquivariance types and analyzes error bounds in more general setup.
Symmetrie Representation Learning. Various works have proposed learning symmetric representa- tions, using transforming autoencoders [20], restricted Boltzmann machines [50]. and equivariant descriptors [47]. particular, [30] shows that convolutional neural networks implicitly learn rep- resentations that are equivariant to rotations, flips, and translations, suggesting that symmetric representations are important inductive biases. Other works have considered learning symmetry- aware features using disentanglement [43], projection mapping [25], equivariance constraints [35], separation into invariant and equivariant parts [61] or subgroups [34]. Park et al. [42] propose learning symmetric encoder that maps to equivariant features and Dangovski et al. [13] learn features that are sensitive and insensitive to different group representations. Other works assume no prior knowledge of symmetry and learn it from data [3, 64, 14, 40J. In particular, Moskalev et al. [40] estimate the difference between the true latent symmetry and the learned symmetry. Similarly, our work considers gap between the true symmetry and model symmetry and theoretically analyze its effects error.
Theory of Equivariant Learning. There are several lines of work on the theory of equivariant learning. Kondor and Trivedi [27] prove that convolutions are sufficient and necessary for equivariance of scalar fields on compact groups, later generalized to the steerable case by Cohen et al. [11]. Certain equivariant networks have been proved to be universal in that such networks can approximate any G-equivariant function [36, 62). Another line of work has considered equivariant networks in terms of generalization error. Abu-Mostafa [1] show that invariant model has VC dimension less than

---

or equal to that of non-equiie equivarian model. Other works studied the generalization error of invariant classifiers by decomposing the input space [51, 46]. Elesedy and Zaidi [17] quantify a generalization benefit for equivariant linear models using the notion of symmetric and anti-symmetric spaces. A PAC Bayes approach was used for generalization bounds of equivariant models [5, 33]. Our work complimentary to these and quantifies approximation error for equivariant model classes.
Data Augmentation. Some methods use data augmentation [29, 28] to encourage the network to learn invariance with respect to transformations defined by the augmentation function [8]. Recent works have explored casss-specfiie [19, 44] and instacce-ppecific [38] data augmentation methods to further boost training by avoiding the potential error caused by uniform augmentation function. Those methods can be viewed as applying data augmentation where pointwise correct or extrinsic
invariance exist, while avoiding incorrect invariance.
Preliminaries
Problem Statement. Consider function f: Y. Let
be the probability density
function of the domain X. We assume that there is no distribution shift during testing, i.e., p is always the underlying distribution during training and testing. The goal for model class {h X Y} to fit the function f by minimizing an error function err(h). We assume the model class {h} is arbitrarily expressive except that it is constrained be equivariant with respect group G. Let 1 be an indicator function that equals to 1 if the condition is satisfied and O otherwise. In classification, err err(h) ) the classification error rate; for regression tasks, the error function a L2 norm function, erTreg (h) Er [1)(()))) (())
erfcls (h) Ex-p (f(z)) * hxx)));
(1)
Equivariant Function. A function f : - Y is equivariant with respect to a symmetry group it commutes with the group transformation g E G, f(gr) gf)),, where acts X through the representation px (g): g acts on E through the representation (g
3.1 Correct, Incorrect, and Extrinsic Equivariance.
Consider a model h which is equivariant with respect to group G. Since real-world data rarely exactly conforms model assumptions, in practice there may often be gap between the symmetry of the model and the ground truth function. Wang et al. [58] propose three-way way classification which describes the relationship between the symmetry of f and the symmetry of h. In this system, h has correct equivariance, incorrect equivariance, or extrinsic equivariance with respect to f.
Definition 3.1 (Correct Equivariance). For all X,9 E G where p(x) if p(gx) 0 and ((x) gJ (x)), J has correct equivariance with respect to f. Definition 3.2 (Incorrect Equivariance). If there exist x E X,g E G such that p(z) > 0, p(gr > 0), but fgz) * (x), à has incorrect equivariance with respect to f.
Definition 3.3 (Extrinsic Equivariance). For all E X,g E G where p(x) > 0, if p(gz) = 0, h has extrinsic equivariance with respect to f.
Example 3.4. Consider binary classification task where X = R and Y = {0, 1}. If the model invariant reflection group G where the group element g E acts on x E X by gr Ø. Figure shows examples when correct, incorrect, or extrinsic equivariance is satisfied.
3.2 Pointwise Equivariance Type.
Although Definitions 3.3 are self-contained. they do not consider the mixture of different equivariance types in single function. In other words, equivariant model can have correct, incorrect, and extrinsic equivariance different subsets of the domain. To overcome this issue, we define pointwise correct, incorrect, and extrinsic equiv- ariance, which is generalization of the prior work.
Incorrect
Rei # f(gz)
f00
y
p(A)
Extrinsic
Correct
plaxi
Figure 2: Example of pointwise correct, incorrect, and extrinsic equivariance in a binary classification task. f(x:) is in black and p(x) is in orange. G is re- flection group that negates x.
Definition 3.5 (Pointwise Correct Equivariance). For g E G and x E X where p(x) * 0. if p(gæ) of 0 and f(gx) (x)), h has correct equivariance with respect to f at x under transformation
3

---

Definition 3.6 (Pointwise Incorrect Equivariance), For 9 E G and z E X where p(r) # 0, p(gr) * 0 and f(gx) * gf(x), h has incorrect equivariance with respect f x under
transformation g-
Definition 3.7 (Pointwise Extrinsic Equivariance). For g E G and a E X where p(x) o 0, if P(gæ) = h has extrinsic equivariance with respect to f at æ under transformation g-
Notice that the definitions of pointwise correct, incorrect, and extrinsic equivariance are mutually exclusive, ie., pair (z. g) can only have one of the three properties. The pointwise definitions are generalizations of the global Definitions 3. 3-1-3 3.3. For example, when pointwise correct equivariance holds for all x X and g G, Definition 3.1 is satisfied.
Example 3.8 (Example of Pointwise Correct, Incorrect, and Extrinsic Equivariance). Consider the same binary classification task in Example 3.4. Figure 2 shows f(x,, g(x), and four subsets of X where pointwise correct, incorrect, or extrinsic holds. For in the correct section (green), p(x)) > O, p(gz) > 0, fz) = fgr)). For x in the incorrect sections (red), p(a) > 0. 0.p(g) O, f(x) + [gx). For x in the extrinsic section (blue), p(a) > O, p(gz)) -0
Definition 3.9 (Correct, Incorrect, and Extrinsic Sets). The Correct Set C x G is subset of X G where pointwise correct equivariance holds for all (z.g) E C. Similarly, the Incorrect Set and the Extrinsic Set E are subsets where incorrect equivariance or extrinsic equivariance holds for all elements in the subset. Denote U S X x G as the Undefined Set where V(z, g) E UU, = 0. By definition we have X G C where II denotes disjoint union.
4 Approximation Error Lower Bound from Incorrect Equivariance
Studying the theoretical error lower bound of an equivariant network is essential for model selection, especially when incorrect equivariance exists. Wang al. [58] prove error lower bound for an incorrect equivariant network, but their setting is limited to a classification task in the global situation of Definition 3.2 with a discrete group and an invariant density function. In this section, we find the lower bound of err(h) for an equivariant model h in a general setting. To calculate such lower bound, we first define the fundamental domain F' of X. Let be the dimension of a generic orbit of G' in X and n the dimension of X. Let v be the (n d) dimensional Hausdorff measure in X.
Definition 4.1 (Fundamental Domain). A closed subset F of X is called a fundamental domain of G in X if X is the union of conjugates' of F, i.e., X Uyea FF, and the intersection of any two conjugates has 0 measure under
We assume further that the set of all x which lie in any pairwise intersection Ug, F#22F (g1 F n F) has measure 0 under v. Let Gx = {gz g E G} be the orbit of æ. then X can be written as the union of the orbits of all points in the fundamental domain F' as such X = UZEr Ga.
4.1 Lower Bound for Classification
We first show the lower bound of the error erT (h) (Equation 1) given the invariant constraint in h: h(gr) h(x).g g E G. In this section, the codomain Y of f is finite set of possible labels. Since h is G-invariant, h has the same output for all inputs in an orbit Gr. We call the label that causes the minimal error inside the orbit the majority label?, and define the error in the orbit as the total dissent.
Definition 4.2 (Total Dissent). For the orbit Gx of J. E X, the total dissent k((ar) is the integrated probability density of the elements in the orbit Ga having different label than the majority label
min / p(=)1(f(2) # v)d2.
MGGz)
yEY JGE
We can also lift the integral to G itself by introducing a factor a(x, g) to account for the Jacobian of the action map and size of the stabilizer of x. (See Appendix A.)
k k(Gx) = min / p(gx))((((9)) + a)a(x,g)dg
(3)
yEY
A conjugate F is defined as gF= {ga|a E F). The majority label has more associated data than all other labels, but does not need be more than 50%.

---

Theorem 4.3. err(h) is lower bounded by k(Gx)dxx
Proof. Rewriting the error function of Equation 1, we have
err(h) p(=) h(x))dr h(r))d JIEF ZEGr
p(=) + h(=))dzde,
(4)
using iterated integration (Appendix B) and Definition 4.1. We assume the measure of F FngFi F O. Since h(z) can only have single label in orbit Gx.
lower bound the inside integral
f JEGG p(z)A(f(2) # b()))dz 2 min / p(=)1(f(2) # y)dz = k(Gæ).
yEY ZEG=
We obtain the claim by integrating over F, Notice that this is a tight lower bound assuming universal approximation. That is, there exists h which realizes this lower bound.
We can express the total dissent in terms of the Incorrect Set I (Definition 3.9). Proposition 4.4. kGGx) e So (1(g E T)a(a', :9)dg, where (Gx)+ (zu Gxlp(ao)) > 0}.
Proof. Consider Equation 3, since the minimum over is obtained for = f(x') for some x' E Ga such that p(z') 0 (i.e., x' (Gx)",
k(Grr) = min Ic p(gx)) (([())) + [(z (2))))(z)))d
eGGr+
Since x' E Ga, then Gx' = Ga and have k(Gx) = k(Gx'). Thus,
1 ((f'))(((gzz') # f(z')) g)dg
k(Gz)) = min
x'e(Gz)+
[ p(gx')1((<', T)(('' .g)d.
min d'E(Gz)
Example 4.5 (Lower bound example for binary classification task using Proposition 4.4).
Let f: + {0, }} be a binary classification function on X
Orbil
Majority Label Ratio
{æo, 22, 233. Let G = C2 = r} be the cyclic group of order two that permutes the elements shows X. the label for each x E X, and how e.r G acts on E X. {xg. 23} forms a fundamental domain F, and there are two orbits: GI {.p, and Ga = {z2, 4z). Since both X and G are discrete and g E acts on X through permutation, The lower bound can be written as err(h) W
X. Figure 3
plaat
04
I
Orait
T Tames Lowrl plmi)
C
Connect Putio
Meal -01 F
p-0
v-1
#nvariant
Figure 4: An exam-
Egec p(gx')
Figure 3: An example
ple multi-class classifi- cation task. Color indi- cates the label. The fun-
I). We can then calculate
binary classification task.
Eger ((ge()()))(((" E T) for x' E X: wo 0.4, 21 0.3, 0, 0. Tak- ing the min over each orbit we have k(Gr,) 0.3, k(G2)) O. Taking the sum over F To, F3) we obtain err(h) 2 0.3.
The circles are elements of X. The arrows show
damental domain F is vertical line. For point F, the orbit Ga is a horizontal line.
how E G acts on æ E X. The arrow color shows whether (ær, g) E I. Example 4.6 (Lower bound example for multi-class classification task using Proposition 4.4). Consider multi-class classification task f R² + Y with = |Y| classes. For x = (u, v) [0, 1] and otherwise p(u, v) 0: i.e., the support
unit square. Let denote the group of translations in the u-direction and h G-invariant network. In a data distribution illustrated in Figure 4, we compute the lower bound for err(h). Consider fundamental domain F (brown line in Figure 4). In the blue area, there is one label across the orbit (i.e., the horizontal
then p(u, v)
5

---

line), meaning Ag E G, (x), C, yielding Proposition 4.4 equals O. For points in the yellow area, the majority label is yellow. This means that for g E G such that g2 in yellow, (x,9) E C; for other G, (x. g) E T. Consequently, Proposition 4.4 is equivalent to the combined green and pink lengths. Taking the integral over F (Theorem 4.3), the lower bound equals the green and pink area (I in Figure 4). We define correct ratio (c) the blue area's s height and majority label ratio (m) as the yellow area's length. Adjusting с and m transitions incorrect to correct equivariance, leading to err(h) 2 area (I) (1 - c) x (1 m.) Appendix H.2 shows experiment where the empirical result matches our analysis.
Lower Bound When G is Finite and The Action of G is Density Preserving. In this section, we consider the lower bound in Theorem 4.3 when G finite and the action of density preserving, y be subset of Gz with label y. Define ((x) = (max,ey (Ga) (lGz)))//l Gz|, which is the fraction of data the orbit Gæ that has the majority label. Denote Q (2())
Le.. p(gx) p(z). Let (Gx), y {2 Gz]f(z)
} the set of all possible values for Q. Consider a partition of
X = II Xg where Xg = fx E ((x) = q}. Define cg = P(x E Xg) = |X
Proposition 4.7. The error lower bound err(h) 2 1 Eg 46g from Wang et al. [58] (Proposition 4.1) is a special case of Theorem 4.3.
Proof in Appendix C. The proposition shows Theorem 4.3 is strict generalization of [58, Prop 4.1].
4.2 Lower Bound for Invariant Regression
In this section, we give lower bound of the error function erTzeg (h) (Equation 1) in regression task given that h is invariant, i.e., h(gr) h(x) for all g G. Assume Y R". Denote by
p(Gz) fzeaz p(z)dz the probability of the orbit Gr. Denote by q(z) = p(=) probability density of the orbit Ga such that Sc., 9(2)dz 1. Let EG [f] be the mean of function on the orbit Gæ defined, and let Va=lf] be the variance of f on the orbit Ga,
p(c=)) the normalized
EG [f] = { ((=){(=)dz Sce (c))())(())
Vo-ls== / [[]lB
Go
((=)(2
Theorem 4.8. err(h) fr p(G=)Waulfldz.
Proof. The error function (Equation 1) can be written as:
err(h) = p(x))l((() h()) J.C [
zear p(=)]I[(z) - ((=)]
Denote e(x) = San p(=)1[((=)> 122). Since G-invariant, there exists e R" such that h(z) = c for all 2 E Gx. Then e(x) can be written as e(x) = Sax p(=)][f(<) - cl3dz. Taking the derivative of e(x) with respect to e and setting it to gives c", the minimum of e(x). Lea (=)) Jos p(=)dz
EGzIf]. Substituting into e(x) we have
e(x) 2 [ p(Gr)
p(z) [a [ʃ] [(=) = P(Ga))aalf]
p(Gx))
We obtain the claim by taking the integral of e(x) over the fundamental domain F.
4.3 Lower Bound for Equivariant Regression
We now prove a lower bound for err(h) ) in a regression task given the model is equivariant, that is. h(px ())((g)))
py (g)h(z) where g E G, px and py group representations associated with X and Y. We will denote Px(g)) and (g)y by gx and gy, leaving the representation implicit. Assume Y g" and a(x, g) is the same in equation 3. Let Id be the identity. Define matrix QGx E IR x and ((gx) E Raxn so that Ic ((gx)(dd = Id by
QG2 = plox)pr (g)T (y (r())a(z,
<(gz) = Qo. (g) (g)a(())))) (5)

---

[()) (2.8)
M(a
[-4,0)
[in)) - (-12)
Riiell Mane
4L-1R0 O
M(xe) (4.4)
Rad) (-8,-2)
O
Ma) (-2,11)
O (lil
p' f1as) 14,
h(2s)
(9)
(d)
Figure 5: An example regression task. (a) The value of f f(x) and the transformation rule (purple)
with respect to group G C4 for all E X. The four points belong to single orbit. (b) When using an invariant network, the minimal error (red) is obtained when the invariant network outputs the mean value (green) of the orbit. (c) For equivariant network, the minimizer (green) can be obtained by taking the mean of the G-stabilized (xx) (inversely transformed) (blue) for all æ in the orbit with respect the transformation rule in the orbit. (d) The minimal error of an equivariant network.
Here, for simplicity, we assume Qc is an invertible matrix. (See Appendix D for general case). If f is equivariant, 9 g (ggr) is constant for all g E G. Define [f]
Ealf
(6)
Theorem 4.9. The error of h has lower bound err(h) 2 Po gEalf, = g)ddd.
See Appendix D for the proof. Intuitively, Eclf x| is the minimizer obtained by taking the mean of all inversely transformed fx) for all æ in the orbit, see Figure 5cd and Example 4.11 below.
SGz p(z)dz. Denote qu : gg- ++ g(gx). Define G-stabilized f as g + g f(zz). When py is an orthogonal representation py G O(n) c GL(n), qu is probability density function on G. Denote the variance of fo as Valful where g ~ qz. The error has a lower bound (hr(h) 2 f5 p(Gx)V Palfadd.
Corollary 4.10. Denote p(Gx)
See Appendix E for the proof. Notice that Corollary 4.10 is generalization of Theorem 4.8. That is, Theorem 4.8 can be recovered by taking (g Id (See the proof in Appendix F).
Example 4.11 (Lower bound example of regression task). Consider regression problem where X f 21, 22, Vz) and Y = R². Assume is uniform density. The cyclic group G = Ca = (e,r, (where e 0 rotation and
T/22 rotation) acts X through iı rT 22 TX1;33 = rT2;; To = r23 (i.e., there is only one orbit Ga = X). g E G acts on E Y through (g( = ( sing cos ). Figure 5a shows the output of f(x), Vx E X. First, consider a G-iivariant
g sin
network h. Since there is only one orbit, Theorem 4.8 can be simplified as: err(h) > Vx the variance of f over X. This can be calculated by first taking the mean of f(x) then calculating the mean square error (MSF) from all J. to the mean (Figure 5b). Second, consider G-equivariant network h. Since
discrete, ga permutes the order of X, py is an orthogonal representation, and there is only one orbit, Corollary 4.10 can be written as err(h) 2 Va Vz[[z), the variance of G-stabilized f. First, to calculate E.(fa], let x = xo. we stabilize from f by g -- f(gx) for all g E G, then take the mean (Figure 5c). We can then find Vo]fa] by calculating the MSE between fxx and transformed mean [fa]fa] (Figure 5d). Appendix H.3 shows an experiment this example's 's environment.
Harmful Extrinsic Equivariance
Wang et al. [58] demonstrate that extrinsic equivariance, where the symmetry imposed on the model data with respect to the input distribution, can lead to a higher performance on the original training data. In this section, argue that this is not necessarily true in all cases, and there can exist scenarios where extrinsic equivariance can even be harmful to the learning problem.
leads to
Consider binary classification task where the domain is discrete and contains only set of four points c TR3, and their labels are either [-1,+1} as shown in Figure 6a. We consider the probability density p to be uniform for this domain, i.c., p(z) 1/4 for the four points S, and p 0 elsewhere.
7

---

This domain is used for both model training and testing so there is no distribution shift. We consider two model classes, Fx. the set of all linear models, and FE. the set of all linear models which are invariant with respect to the cyclic group C2 {1,g), where g( 22, 2t) = (x1, 22, -2a). FN corresponds to an unconstrained or a non-equivariant model class and FE corresponds to an extrinsically equivariant class for this domain. For the labeling shown in Figure 6, the hyperplane x3 = correctly classifies all samples and is contained in FN. However, a function fe E FE is equivalent to linear classifier on 8² and effectively sees the data as Figure 6b". This exclusive-or problem does not admit linear solution (it can be correct for at most points).
Concretely, we can compute the empiri- cal Rademacher complexity, a standard mea- sure of model class expressivity, for non- equivariant and extrinsically equivariant model classes and show that FE has lower com- plexity than FN- Recall that empirical Rademacher complexity is defined as Rg (F)= E, sup 1 E fx()), where S is the set of samples and a (d1, (01...., mm) , E
R
x
4/3
X;
(a) Data
(b) Transformed (e) Cz-equivariant view
data
Figure 6: An example dataset where extrinsic equivariance increases the problem difficulty. The samples are of the form I = (2), Z2, F3) and the labels are shown as different shapes. A C2- equivariant linear model transforms the original data (a) into (b), which is equivalent to viewing the data as in (c). The original task has an easy solution (e.g. hyperplane at z3 0), while the C2- invariant view the classic exclusive-or problem.
{-1,+1} are independent uniform Rademacher random variables, and x' is the i-th sample. As there exists some linear function fn E FN that fully classifies S for any combination of labels, {F(Fx) = 1. For the extrinsic equivari- ance case, of the 16 possible label combinations, there are two cases where fe E Fg can at most classify out of 4 points correctly, and thus
RsFFF) = 32 < Rs(FN) (see Appendix G for the calculations). This illustrates that in certain cases, extrinsic equivariance can lead to lower model expressivity than no equivariance and thus be harmful to learning.
31
Experiments
We perform experiments validate our theoretical analysis on both the lower bounds (Section 4) and the harmful extrinsic equivariance (Section 5). We find that our bounds accurately predict empirical model error. In addition the experiments in this section, Appendix H.2 shows an experiment verifying our classification bound (Theorem 4.3) and Appendix H.3 shows an experiment verifying our regression bound (Theorem 4.8 and 4.9). The experiment details are in Appendix I.
6.1 Swiss Roll Experiment
We first perform an experiment a vertically separated Swiss Roll data distribution, see Figure 7 This example, similar to that in Section 5, demonstrates that Cz-invariant model effectively "flattens" the z-dimension of the data so must learn the decision boundary between two spirals (Figure 7b), whereas the non-equivariant model only needs to learn horizontal plane to separate the classes, a significantly easier task. Besides the extrinsic data distribution, we consider two other data distributions shown in Figure 7c and Figure 7d, where C2-invariant model will observe incorrect and correct equivariance due to the mismatched and matched data labels the two = planes.
We combine data from all three distributions in various proportions to test the performance of a z-invariant network (INV) with baseline unconstrained network (MLP). Let e be the correct ratio, the proportion of data from the correct distribution. Define the incorrect ratio i and extrinsic ratio e similarly. We consider all c, i, e that are multiples of 0.125 such that c + + e = 1. Figure 7ef shows some example data distributions. Relative to INV, this mixed data distribution has partial correct, incorrect, and extrinsic equivariance, which not fully captured in prior work [58]. Based on Proposition 4.4, we have k ((Gr)) = 0.5 for drawn from the incorrect distribution, and ((G = 0 otherwise. Since the data is evenly distributed, we can calculate the error lower bound err(h) 2 0.5i
Notice that the four additional points Figure 6b compared with (a) are not the domain, they are created through applying the transformation rule Fg onto the domain. For visualization, show simpler version of the data distribution. See Appendix I. for the actual one.
8

---

(a) Swiss Roll Data
(b) C2-Invariant
(c) Incorrect
(d) Correct
(c)
== 5,
(f) = 5,
View
5 0
0 5
Figure 7: (a) (b) The Swiss Roll data distribution that leads to harmful extrinsic equivariance. (c) (d) The correct and incorrect data distribution in the Swiss Roll experiment. Here the spirals overlap with mismatched and matched labels respectively. (e) (f) Data distribution example with different correct ratio (c), incorrect ratio (i), and extrinsic ratio (e) values.
Results. Figure 8a shows the test success rate of INV compared with MLP when and vary with = 0. When increases, the performance of IN' decreases while the performance of
MLP shows an inverse trend, demonstrating that extrinsic equivariance is harmful in this ex- periment. Figure shows the performance of INV and MLP when and vary while = O. The green line shows the upper bound of the test success rate (1 0.5i). The experimen- tal result matches our thcoretical analysis quite closely. Notice that when increases, there a bigger gap between the performance of the network and its theoretical upper bound, since classification in the correct distribution a harder task. Appendix H.1 shows the com- plete results of this experiment.
RV LB
coneel
(a)
(b)
Figure 8: Result of the Swiss Roll experiment. (a) test success rate of an invariant network (red) and an unconstrained MLP (blue) with different extrinsic and correct ratio when incorrect ratio is O. (b) same as (a) with different correct and incorrect ratio when extrinsic ratio is O. Averaged over 10 runs.
6.2 Digit Classification Experiment
In this experiment, we apply our theoretical analysis realistic digit classification task using both the printed digit dataset [16] and the MNIST handwritten digit dataset [15]. We compare a Da-invariant network (D4) with an unconstrained CNN. In the printed digit classification, D4 exhibits incorrect equivariance for and 9 under a æ rotation. Using Theorem 4.3, we can calculate lower bound of error for D4 10%. However, as shown in Table (top), the experimental results indicate that the actual performance is slightly better than predicted by the theory. We hypothesize that this discrepancy arises because rotated 9 differs slightly from a in some fonts. We conduct a similar experiment using the MNIST handwritten digit dataset (Table I bottom), where D2 achieves even better performance in classifying 6 and 9, This improvement is likely due to the more distinguishable handwriting of these digits, although the performance still underperforms the CNN as incorrect equivariance persists. important to note that there is a significant decrease in performance for D4 when classifying 2/5 and 4/7 compared to the CNN. This is because vertical flip results in incorrect equivariance when classifying handwritten 2/5, and a similar issue arises for 4/7 under T/2 rotation followed by vertical flip (notice that Weiler and Cesa [60] make similar observation). These experiments demonstrate that our theory is useful not only for calculating the performance bounds of an equivariant network beforehand, but also for explaining the suboptimal performance of an equivariant network, thereby potentially assisting model selection.
6.3 Robotic Experiment
In this experiment, we evaluate our theory in behavior cloning in robotic manipulation. We first preform an experiment where the problem is a mixture of correct and incorrect equivariance for Dj -reuuiivai policy network (D,) where the robot's action will flip when the state is flipped

---

Digit
Overall
0
5
6
7
8
9
Print CNN
96.62
99.81
98.27
97.06
96.12 12
98.04
92.9
94.93
97.85
93.65
96.13
Print D4
92.5
99.71
98.62
96.94
95.98
97.91
93.52
63.08
98.42
95.88
76.17
Print Upper Bound
90
100
50
MNIST CNN
98.21
99.51
99.61
98.62
98.83
98.08
98.47
97.99
97.04
96.98
96.81
MNIST D4
95.15
98.93
99.21
91.84
98.28
95.49
95.04
93.71
95.67
97.73
95.34
Table I: D4-invariant network compared with unconstrained CNN in printed and MNIST hand-
written digit classification tasks. Bold indicates that there is a 1% difference two models.
Stacking with Correct Equi.
Borting with Incoreet Equi. Sorting with Extrinsic Equi.
Probability of Stacking (correct equiit
Probabiitty Sarting (ncorrect equi.:
Calar 4f
Duto 0189
03.0
CNN
cornec
Figure 9: Left: environment containing both correct (Stacking) and incorrect (Pushing) equivariance for a Di-equivariant (horizontal flip) policy net. Right: environment with harmful extrinsic equivariance for the same policy.
Figure 10: Result of robotic experiment with different correct ratio. Averaged over runs.
horizontally. Specifically, the environment contains two possible tasks (Figure 9 left). Stacking requires the robot to stack a green triangle on top of a blue cube. Here flip equivariance is correct. Sorting requires the robot to push the red cube to the left and the yellow triangle to the right. Here flip equivariance is incorrect because the robot should not sort the objects an opposite way when the state is flipped (in other words. D1 cannot distinguish left and right). We vary the probability e of the stacking task (correct ratio) in the task distribution, and compare the performance of D1 versus standard CNN policy. If we view the sorting task as a binary classification task, we can caleulate an upper bound of performance for D using Theorem 4.3: 0.5 0.5c. Figure 10 shows the result. Notice that the performance of D closely matches the theoretical upper bound, while the performance of CNN remains relatively stable for all Stacking-Sorting distributions.
We further evaluate D1 in sorting task with harmful extrinsic equivariance. Here, the goal for the robot is the same as sorting above (push the red cube left and the yellow triangle right), however, the left and right sides of the workspace can now be differentiated by gray-scale colors. The shades of gray are discretized evenly into bins, where the left side's color randomly sampled from the odd-numbered bins, and the right side's color is randomly sampled from the even-numbered bins (Figure right). The different color distributions of the left and right sides make D2 extrinsically equivariant, but it needs to learn the color distribution to distinguish left and right (while CNN can distinguish left and right directly). We set n = 10, and D achieves 71.5 1.6% test success rate, while CNN achieves 99.5 + 0.5%, demonstrating that the Dj extrinsic equivariance is harmful in this task. See Appendix I.5 for the details of the robot experiment.
7 Discussion
This paper presents a general theory for when the symmetry of the ground truth function and equiv- ariant network are mismatched. We define pointwise correct, incorrect, and extrinsic equivariance, generalizing prior work [58] to include continuous mixtures of the three extremes. We prove error lower bounds for equivariant networks applied asymmetric tasks including classification, invariant regression, and equivariant regression without the assumption of invariant data density. Our work discusses the potential disadvantage of extrinsic equivariance, and provides experiments that validate our theoretical analysis. The major limitation of this paper is that our theoretical lower bounds require domain knowledge like the density function over the domain. future work, we will develop easy-to- apply model selection tools using our theory. Another future direction is theoretically understanding when extrinsic equivariance helpful harmful and analyzing the effect of extrinsic equivariance on the decision boundary of equivariant network.
10

---

Acknowledgments
This work supported in part by NSF 1724257, NSF 1724191, NSF 1763878, NSF 1750649, NSF 2107256, NSF 2134178, NSF 2312171, and NASA 80NSSC19K1474,
References
[1] Yaser S Abu-Mostafa. Hints and the dimension. Neural Computation, ((2):27--28.. 1993.
[2] Brandon Anderson, Truong Son Hy, and Risi Kondor. Cormorant: Covariant molecular neural networks. Advances in neural information processing systems, 32, 2019.
[3] Fabio Anselmi, Georgios Evangelopoulos, Lorenzo Rosasco, and Tomaso Poggio. Symmetry- adapted representation learning. Pattern Recognition, 86:201-208, 2019.
[4] Kenneth Atz, Francesca Grisoni, and Gisbert Schneider. Geometric deep learning on molecular representations. Nature Machine Intelligence, 3(12):1023-1033 2021.
[5] Arash Bchboodi, Gabriele Cesa, and Taco Cohen. A pac-bayesian generalization bound for cquivariant networks. arXiv preprint 22i:::2210.13100 2022.
[6] Alexander Bogatskiy, Brandon Anderson, Jan Offermann, Marwah Roussi, David Miller, and Risi Kondor. Lorentz group equivariant neural network for particle physics. In International Conference on Machine Learning, pages 992-1002. PMLR, 2020.
[7] Evangelos Chatripantazis, Stefanos Pertigkiozoglou, Edgar Dobriban, and Kostas Daniilidis. -qquivaaaa attention networks for shape reconstruction in function space. In The Eleventh International Conference on Learning Representations, 2023. URL https::/ //tp::///ppeeevier. net/forumiid RDD3DDDbjjjj
[8] Shuxiao Chen, Edgar Dobriban, and Jane Lee. group-theoretic framework for data augmentation. The Journal of Machine Learning Research, (111)))988599559 2020.
[9] Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference machine learning, pages 2990-2999. PMLR, 2016.
[10] Taco S. Cohen and Max Welling. Steerable CNNs. In International Conference on Learning Representations, 2017. URL /ttpss//openrevieee wtwperrwwwwwww.eeeeeeeeeeeeeeeee net/fforum?id-rrQKYY.
[11] Taco Cohen, Mario Geiger, and Maurice Weiler. A general theory of equivariant cmns on homogeneous spaces. Advances in neural information processing systems, 32, 2019.
[12] Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. http://pybulle. org, 2016-2021.
[13] Rumen Dangovski, Li Jing, Charlotte Loh, Seungwook Han, Akash Srivastava, Brian Che- ung, Pulkit Agrawal, and Marin Soljacic. Equivariant self-supervised learning: Encouraging cquivariance representations. In International Conference on Learning Representations, 2021.
[14] Nima Dehmamy, Robin Walters, Yanchen Liu, Dashun Wang, and Rose Yu. Automatic symmetry discovery with lie algebra convolutional network. Advances in Neural Information Processing Systems, 34:2503-2515, 2021.
[15] Li Deng. The mnist datahase of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):11-1422 2012.
[16] Kshitij Dhama. Printed numerical digits image dataset. https://github. wow.aubb..com//yydee00022222 -iinted-diiits-. dataset, 2021. GitHub Repository.
[17] Bryn Flesedy and Sheheryar Zaidi. Provably strict generalisation benefit for equivariant models.
In International Conference Machine Learning, pages 2959-2969, PMLR, 2021.
11

---

[18] Mare Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data, In International Conference on Machine Learning, pages 31 3165-3176 PMLR, 2020.
[19] Søren Hauberg, Oren Freifeld, Anders Boesen Lindbo Larsen, John Fisher, and Lars Hansen. Dreaming more data: Class-dependent distributions over diffeomorphisms for learned data augmentation. In Artificial intelligence and statistics, pages 342-350. PMLR, 2016.
[20] Geoffrey E Hinton, Alex Krizhevsky, and Sida D Wang. Transforming auto-encoders. In International conference on artificial neural networks, pages 44-51. Springer, 2011.
[21] Haojie Huang, Dian Wang, Robin Walters, and Robert Platt. Equivariant transporter network. In Robotics: Science and Systems, 2022.
[22] Haojie Huang, Dian Wang, Xupeng Zhu, Robin Walters, and Robert Platt. Edge grasp network:
A graph-based 3E(3)-invariai) approach to grasp detection. In International Conference Robotics and Automation (ICRA), 2023.
[23] Mingxi Jia, Dian Wang. Guanang Su, David Klee, Xupeng Zhu, Robin Walters, and Robert Platt. Seil: Simulation-auggenntn equivariant imitation learning. In International Conference on Robotics and Automation (ICRA), 2023.
[24] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[25] David Klee, Ondrej Biza, Robert Platt, and Robin Walters. 12i: Image to icosahedral projection for SO(3) object reasoning from single-view images. arXiv preprint arXiv:2207.08925 2022.
[26] David Klee, Ondrej Biza, Robert Platt, and Robin Walters, Image sphere: Learning equivariant features for efficient pose prediction. In The Eleventh International Conference on Learning Representations, 2023. URL https:/ //ppnreiie//eeeiieeee wttppeerreieee..tttttrrrreeee wwt.//euum/miiiie idd=
http:/oorrrrr............... _22DpAtr7PP.
[27] Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In International Conference on Machine Learning, pages 2747-2755. PMLR, 2018.
[28] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[29] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applicd to document recognition. Proceedings of the IEEE, (6(11):2278-23) 1998.
[30] Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equivariance and equivalence. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 991 991-999, 2015.
[31] Xiaolong Li, Yijia Weng, Li Yi, Leonidas J Guibas, A Abbott, Shuran Song, and He Wang. Leveraging se (3) equivariance for self supervide category- object pose estimation from point clouds. Advances in Neural Information Processing Systems, 012115770-138811 2021.
[32] Xueyi Liu, Ji Zhang, Ruizhen Hu, Haibin Huang. He Wang, and Li Yi. Self-supervised category- level articulated object pose estimation with part-level SE(3) equivariance. In The Eleventh International Conference on Learning Representations, 2023. URL https://opennreviex net/forum?id 26Gtt
[33] Clare Lyle, Mark van der Wilk, Marta Kwiatkowska, Yarin Gal, and Benjamin Bloem-Reddy. On the benefits of invariance in neural networks. arXiv preprint arXiv:2005.00/78, 2020.
[34] Kaitlin Maile, Dennis George Wilson, and Patrick Forré. Equivariance-aware architectural optimization of neural networks. In The Eleventh International Conference on Learning Representations, 2023. URL hattps://openreeee wwwprrvviiieeeeeeforrrreee met//forumPdd-aGraaf..e
[35] Giovanni Luca Marchetti, Gustaf Tegnér, Anastasiia Varava, and Danica Kragic. Equivariant representation learning via class-pose decomposition. arXiv preprint a:Xiv:2007.03116, 2022.
12

---

[36] Haggai Maron, Ethan Fetaya, Nimrod Segol, and Yaron Lipman. On the universality of invariant networks. International conference on machine learning, pages 4363-4377 PMLR, 2019.
[37] Haggai Maron, Or Litany, Gal Chechik, and Ethan Fetaya. On learning sets of symmetric elements. In International conference on machine learning. pages
PMLR, 2020.
[38] Ning Miao, Tom Rainforth, Emile Mathieu, Yann Dubois, Yee Whye Teh, Adam Foster, and Hyunjik Kim. Learning instance-specific augmentations by capturing local invariances. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 24720-24736. PMLR, 23-29 Jul 2023. URL https: https::/ htpp://////cccce.s...... mlr Salr.press press wttp:///222mmmaamm .tmml.
[39] Arnab Kumar Mondal, Vineet Jain, Kaleem Siddiqi, and Siamak Ravanbakhsh. EqR: Equiv- ariant representations for data-efficient reinforcement learning. In Kamalika Chaudhuri, Ste- fanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceed- ings of the 39th International Conference Machine Learning, volume 162 of Proceed- ings of Machine Learning Research, pages 15908-15926. PMLR, 17-23 Jul 2022. URL https:/ http:///////c//ggg..../. mlr wwwpp/pwsseeeepppp presss/v1622mmoda12222 html.
[40] Artem Moskalev, Anna Sepliarskaia, Ivan Sosnovik, and Arnold W.M. Smeulders. LieGG: Studying learned lie group generators. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL settps://openreviee wttp://.eeeeeeeeeeeeeeeee settt/forummidd=sk2200a
[41] Chuer Pan, Brian Okorn, Harry Zhang, Ben Eisner, and David Held. Tax-pose: Task-specific pross-pose estimation for robot manipulation. In Conference on Robot Learning, pages 1783- 1792. PMLR, 2023.
[42] Jung Yeon Park, Ondrej Biza, Linfeng Zhao, Jan Willem van de Meent, and Robin Walters. Learning symmetric representations for equivariant world model. In International Conference on Machine Learning, 2022. URL https: //aa::///axvvvi org/
11371.
[43] Robin Quessard, Thomas Barrett, and William Clements. Learning disentangled representations and group structure of dynamical environments. Advances in Neural Information Processing Systems, 13:79777-9973 2020.
[44] Cédric Rommel, Thomas Moreau, Joseph Paillard, and Alexandre Gramfort. Cadda: Class-wise automatic differentiable data augmentation for eeg signals. International Conference on Learning Representationss 2021.
[45] Hyunwoo Ryu, Hong in Lee, Jeong-Hoon Lee, and Jongeun Choi. Equivariant descriptor fields: -quii-eqqi energy-based models for end-to-end visual robotic manipulation learning. In The Eleventh International Conference on Learning Representations, 2023. URL /ttps://oppereeier: nwt//eeemiidddaeeeeooo ZSPCmY50..
[46] Akiyoshi Sannai, Masaaki Imaizumi, and Makoto Kawano. Improved generalization bounds of group invariant/equivariant deep networks via quotient feature spaces. In Uncertainty in Artificial Intelligence, pages 771-780. PMLR, 2021.
[47] Uwe Schmidt and Stefan Roth. Learning rotation-ave awarre features: From invariant priors to equivariant descriptors. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 2050-2057. IEEE, 2012.
[48] Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua B Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann. Neural descriptor fields: Se (3)-eqiiarriant object representations for manipulation. In 2022 International Conference Robotics and Automation (ICRA), pages 6394-6400. IEEE, 2022.
[49] Anthony Simeonov, Yilun Du, Yen-Chen Lin, Alberto Rodriguez Garcia, Leslie Pack Kaelbling, Tomás Lozano-Pérez, and Pulkit Agrawal. Se (3)-qqivari relational rearrangement with neural descriptor fields. In Conference Robot Learning, pages 835-846. PMLR, 2023.
13

---

[50] Kihyuk Sohn and Honglak Lee. Learning invariant representations with local transformations. In John Langford and Joelle Pineau, editors, Proceedings of the 29th International Conference on Machine Learning (ICMI-12), ICMI. '12, pages 1311-1318, New York, NY, USA, July 2012. Omnipress. ISBN 978- 1-850--
[51] Jure Sokolic, Raja Giryes, Guillermo Sapiro, and Miguel Rodrigues. Generalization error of invariant classifiers. In Artificial Intelligence and Statistics, pages 1094-1103, PMLR, 2017.
[52] Elise van der Pol, Daniel Worrall, Herke van Hoof, Frans Oliehoek, and Max Welling. Mdp homomorphic networks: Group symmetries in reinforcement learning. Advances in Neural Information Processing Systems, 33, 2020.
[53] Robin Walters, Jinxi Li, and Rose Yu. Trajectory prediction using equivariant continuous convolution. arXiv preprint arXiv:2010. 1144,, 2020.
[54] Dian Wang, Robin Walters, Xupeng Zhu, and Robert Platt. Equivariant Q learning in spatial action spaces. 5th Annual Conference on Robot Learning, 2021. URL. https: //openreview
[55] Dian Wang. Mingxi Jia, Xupeng Zhu, Robin Walters, and Robert Platt. On-robot learning with equivariant models. In 6th Annual Conference on Robot Learning, 2022. URL https: /openreview net. wwt.tnttttoriieeee atte//ofommmaaaa000002222. d-K866bbP22yh.
[56] Dian Wang, Robin Walters, and Robert Platt. (Quuieaquiaai reinforcement learning. In International Conference on Learning Representations, 2022. URL https::/opeenreviir
ntt//forum77ae77700hhhrrt.
[57] Dian Wang, Colin Kohler, Xupeng Zhu, Mingxi Jia, and Robert Platt. Bulletarm: An open- source robotic manipulation benchmark and learning framework. In Robotics Research, pages 335-350. Springer, 2023.
[58] Dian Wang, Jung Yeon Park, Neel Sortur, Lawson L.S. Wong, Robin Walters, and Robert Platt. The surprising effectiveness of equivariant models in domains with latent symmetry. In International Conference on Learning Representations, 2023. URL //penrei/per net/forumidd ietoormmiidpaae PaNAduGGAAAAU.....
[59] Rui Wang, Robin Walters, and Rose Yu. Incorporating symmetry into deep dynamics models for improved generalization. arXiv preprint arXiv:2002.03061 2020.
[60] Maurice Weiler and Gabriele Cesa. General (2)-equivariant steerable cnns. Advances in Neural Information Processing Systems, 32, 2019.
[61] Robin Winter, Marco Bertolini, Tuan Le, Frank Noe, and Djork-Amé Clevert. Unsupervised learning of group invariant and equivariant representations. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https: htttps://open/ee http////pp/eeeeeee////e/ee .ww/ htt.//ommiddd711122200
[62] Dmitry Yarotsky. Universal approximations of invariant maps by neural networks. Constructive Approximation, 55(1):407-474, 2022.
[63] Linfeng Zhao, Xupeng Zhu, Lingzhi Kong, Robin Walters, and Lawson L.S. Wong. Integrating symmetry into differentiable planning with steerable convolutions. In The Eleventh International Conference on Learning Representations, 2023. URL /ttps://openreviiee wwwpweiwwwwiiweernneeeeee net/forum? id=n7CPzMPKQL.
[64] Allan Zhou, Tom Knowles, and Chelsea Finn. Meta-learning symmetries by reparameterization. arXiv preprini arXiv:2007.029333 2020.
[65] Xupeng Zhu, Dian Wang, Ondrej Biza, Guanang Su, Robin Walters, and Robert Platt. Sample efficient grasp learning using equivariant models. In Robotics: Science and Systems, 2022.
14

---

A Integrals on the Group
Fundamental Domains In this paper, we are interested in cases in which the group is not necessarily discrete but may have positive dimension. We do not assume the fundamental domain has non-empty interior, and thus domain is misnomer, In this case the conjugates of the fundamental domain gf have measure 0 and the condition that their intersection have measure 0 is vacuous. Instead we assume a stronger condition, that the union all pairwise intersections Ug (gi F n F2 has measure O. We also require that F' and the orbits Gr are differentiable manifolds such that integrals over X may be evaluated fx f(x)dx fr Jay f (z)dzdy similar to Equation 8 from [18].
Reparameterization Consider the integral
Jce. f(z)dz.
Denote the identification the orbit Gæ and coset space G/G, with respect to the stabilizer G2 {g: : gr 2} by az: G/Gz Ga. Then the integral can be written
Jajo. f(gx)
Ouz (g)
dg.
Ög
We also lift the integral to G itself
0a
(1 dh
-1 (1oo dh ) f(gz)
0a.z (g)
JoiG. f(gx)
dğ
dg
0g
Og
(1 dh
-1
S.scc G.
( f(ghx)
(a)(
dhdg
0g
-1 [(c f(gx))
(g)
dh
dg.
Ög
Define a(g,x) (Sc. dh)
(a.(ū)
8g Then
[ Gr
[(=)dz /( [(zz)a(g
B Iterated Integral
Let X be an ne-dimensional space, Definition 4.2 (Equation 2) defines k(Gz) as an integral over Ga C X, which is m-dimensional sub-manifold of X. In Theorem 4.3, Equation 4 rewrites the error function (Equation I) as an iterated integral over the orbit Gr and then the fundamental domain F using Definition 4.1. In the discrete group case, m would be 0, Equation is an integral of O-form in 0-manifold, which is a sum:
kGrr) = min Σ (f)) yEY SEGx
# = min Σ (((g)) + WEY 9EG
(8)
C Proof of Proposition 4.7
Proof. Consider the integral of probability density inside Ga, for a given it can be separated into two groups:
[ p(z)dz=
/ p()]1({(2) = y)d=
GG
p(z)L(f(x) + y)dz. Ga
15

---

We can then rewrite k(Gx) in Equation 2 as:
min 1/c p(z)dz / p(=)1(f(=)= = y)dz WEY
k(Gz)
(9)
G
Letting (Gz)y fo' Gz f(x)) y) f-( n Gz, Equation 9 can be written as:
WEY 1/2 p(z)dz - p(2)dz)
k(Gz) min
max / p(z)dz. p(z)dz
Gz
yEY (Gx),
Theorem 4.3 can be rewritten as:
(f plz)d=
J.cn, p(z)dz) d
err(h) 2
max NEY (Gx),
Gz
/ p(z)dz- - S1 max I J(GE)) r(=)dz F Gzz
wey J max ((Gx) ((G=)),l(()ddd.. DEY
(10)
The first term in Equation 10 uses the fact that = UZEF Gr so the integral of the probability the orbits of all points in the fundamental domain is the integral of the probability of the input domain X which is 1. The second term of Equation 10 uses p(gx) = p(z) so the integration of p(=) on (Gr), becomes p(z) times the range of the limit which is the size of (Gx)y. (Gr),y
Now consider a partition of = [I, Fg where Fg = {ar F (maxzer [G.a) ((Gx),))/|Gtt = q). We
can rewrite Equation 10 as:
err(h) 2 1 / g(Gzdp(()d
(11)
2 1 Σ / d(Cx/(oo)dr
(12)
Fg
9 /// [Grlp(z)dz.
21 Σ
(13)
Equation 11 uses the definition of q. Equation 12 separates the integral over E' into the partition of F. Equation 13 moves & out from the integral because it is constant inside the integral. Consider the definition of Cy. we have:
Cg = F(x E X²)
I p(r)dr
J [ p(z)dzda
(14)
= / F,
Grllp(r)dd..
(15)
Equation uses X, = UsEF, Gr. Equation 15 uses p(x) = p(gx). Now we can write Equation 13
err(h) Σ 9cq.
16

---

D Proof of Theorem 4.9
Define ((gx) _Rxx such that
QGo((gx) p(gx)py (g)² pr(g)a(z,))
(16)
In particular, g(gx) exists when Qa is full rank. follows that Qc2 la dgg QGw Moreover, QG and ((gx) are symmetric matrix.
Proof. The enror function (Equation 1) can be written
err(h) Ex- fllf(())()))))))))))))))))))))) h())l
I p(z))1((x) - h22)1lddx
= JEF J9EG
p(gz))lf(ga) h(gx)]Ba(x,, g)ddr.
Denote e(x) So (( (aar) ()dg. Since is G-equivariant, for each F the value h(x)) Rh of h determines the value of h across the whole orbit h(gx) gh(x) go for E G. Then e(x) can be written
e(x) = p(gx))lf(gx) gclBa(zz, g)dg
p( c)llaa(r.gdd)
(g f(gg))))))))) c)² p(gz)=" ga 66 f(gr) c)dg
(g- - c) Qcun(())( c)dg.
(17)
Taking the derivative of e(x) with respect to we have
de(x)
/ ((Qc))(0)))* + (Qcalloz))) f(gx))
dc
2cc.( (0llc-ggg(((()))))
Setting 0c(x)/Oc 0 find an equation for c* which minimizes e(x)
QGa [ Algr)dg c² QGa / g(gr) Ja
QG QG. Ec(f,x))
(18)
Substituting o* into Equation 17 we have
e(x) 2
(g f(gx)
c") (ool((ocl00)))))) ( c
6
[
QcT f(gx))
(19)
Q f(gr)
+ ( dg.
The term TT - (gr)dg could be simplified
17

---

Jo "C ((xz))
f(gx)dg
Ealf, -
(20)
Notice that QG., and (ggz) are symmetric matrix
Jc Qaal(or)))
/ xh Q(
G
Thus Equation 19 becomes
1 ((="((t))) f(xx))
e(x)
J
(ol T
El = -000aalo0)gg-((1
p(gz))lf(gz) - gEalf x112a(2 g)dg.
Taking the integral over the fundamental domain F we have
err(h) / e(x)
F / p(gx))li(ge)) gEalf, =f g)ddd..
(21)
E Proof of Corollary 4.10
Proof. When py is orthogonal representation, we have Py (g)T pr (g) In, i.e., the identity matrix. Then ((gx) can be written as q(gz) = s(gx) Id where s(gx) is scalar. Since Je d(gx)dg Id, we can re-define q(gz) to drop Id and only keep the scalar, then (g(g) can be viewed as probability density function of because now Jo (g) = 1.
With qu (g) being the probability density function, Eg [f, æ] (Equation 6) naturally becomes the mean Eo[fa] where g ~ q=-
Now consider e(x) = Sc ((ox)))lf((z) - -BB g) in Theorem 4.9, it can written as
<(x)
p - (BCc)))))
- gBa
p(gz))lo(g
f((r)) f(g))))((~'((g)) -ool))))))
(5 -e (r) - (a( g)dg.
Since py ()) PY (g) Ln we have
e(x) p(gx)( ((gx) Rolla)) f(gr) EG(fa))((r.gd4
p(gx))ll-11 ((((x) - EalfallBa(z.))dd
(22)
18

---

From Equation we have p(gr)a(2zg)) 9) = QGzl(gz). Substituting in Equation 22 we have
f(ga) Ealnn]l2dg. e(x) / Qc. ((go)
Since QGx = So ((g when py (g) pr (g) = Ins we have
e(x) QG= / 90(9)19 f(gz) Bclf -QaxVoll
(23)
Now consider Qa. (Equation 5), when pr (g)² pv (g) Ia. it can be written
QG / p(gr)a(zzg)dg
1 p(z)d=
Gz
= p(Gx).
Replacing QG with p(Gx) in Equation 23 then taking the integral of e(x) over the fundamental domain gives the result.
F Lower Bound of Equivariant Regression when py = Id
Proposition F.1. When py = Id, the ervor of has lower bound err(h) 2 Je:(GG))ccllldd
which is the same Theorem 4.8.
Proof. Consider Equation 5, when (r(g) = Id, we have
QG plgz)a(z, gdd..
Exchange the integration variable using
ge we have
QG = = f p(z)dz.
(24)
Gz
Consider Eg[sa] = So ( When Py py Id, becomes
[a[sz] = / ((gx))(grrdd)
Substituting ((g=) with Equation 5, considering Py (g) Id, we have
Eclfal [( (
Exchange the integration variable using 2 gx we have
Eolfs [ (obp(E)f(E)d..
Substituting Equation 24 we have
Eo[sa] = / p(2)
Gz Jan (z))dz f())dz
= -E.
Similarly, proof Volf2] = Voulff, thus when py 1d, Corollary 4.10 is Theorem 4.8. G
19

---

Non-equivariant model class
Cz-equivariant model class
6
sup J.ENN m
Ei=1 osfn(x")
J.EF 1 Ei=1 oefn
1
0.75
1
1
1
1
1
0.75
Rg
1
(a) Correct
(b) Incorrect
(c) Extrinsic
(d)
Extrinsic
Invariant
View
Figure 11: The correct, incorrect, and extrinsic data distribution in the Swiss Roll experiment.
G Rademacher Complexity of Harmful Extrinsic Equivariance Example
Let S = 1 2 3.30000 xx1 where the labels are y y² +1 and y² y² = -1 We consider two model classes FN. the set of all linear models, and FF. the set of all linear models equivariant to C2, and compute their empirical Rademacher complexity on S.
For the data S, an extrinsically equivariant linear model class has lower empirical Rademacher complexity than its unconstrained linear counterpart, demonstrating that extrinsic equivariance can be harmful to learning.
H
Additional Experiments
H.1 Swiss Roll Experiment
Figure 11 and Figure 12 show the actual data distribution for the Swiss Roll experiment in Section 6. In the incorrect distribution, the data in the two planes form two spirals with different labels but the same shape. The equivariance is incorrect because if we translate one spiral the other spiral's sss plane, they will overlap but their labels are different. In the correct distribution, there are two different 'dashed' spirals copied into two z-planes. The equivariance correct because after z-translation, both the data and their labels exactly overlap. In all three cases, we assume the data has a uniform distribution. Figure 13b shows the ternary plot of ML.P for different e, ir, er, where the performance of MLP decreases the correct ratio increases. Figure 13a shows an inverse trend: the
20

---

c= 5,1= 5, 0
(b) c= 5, b =1==0, 0, (c) 0,1 e
(d)
c= 55.5
.25 c .25
Figure 12:
Data distribution example with different correct ratio (c), incorrect ratio (ér), and extrinsic ratio (er) values.
INV
MLP
(a)
(b)
Figure 13: The ternary plot of the invariant network (a) and unconstrained network (b) with different correct, incorrect, and extrinsic ratio.
performance of INV increases as the correct ratio increases. Moreover, both extrinsic and incorrect equivariance harms the performance of INV, but incorrect equivariance is more devastating because the error is limited by a theoretical lower bound.
H.2 Square Experiment
We consider the environment shown in Example 4.6. We vary {0.2, 0.4, 0.6, 13 and e E 0.,, 0.4, 0.6, 0.8, 1}. We train an u-invariant network and evaluate its test performance with the theoretical lower bound err(h) 2 (1 e) (1 - ()) Figure 14 shows the test error of the trained network compared with the theoretical lower bound. The highest difference is below 3%, demonstrating the correctness of our theory.
H.3 Regression Experiment
In this experiment, we validate our theoretical error lower bound for invariant and equivariant
regression (Theorem 4.8, 4.9) in environment similar to Example 4. Consider regression task
f: x - IR2 given by (0, x) + y. where X {5r, Ii U2, 23). The group G = C4 =
{e, 2 r3 acts on (0,x) .)) by g(0, x) (0. through permutation:
rTgi 32 rw1i
Invariant Network
Equivariant Network
Empirical/Thaocciiaal
1.002 +00000
1.001 +=0.000
Table 2: Empirical (h) divided by theoretical err(h) for invariant regression and equivariant regression. Results averaged over 100 runs with different f for each regression. Empirical regression emor matches theoretical error.
21

---

and
25
0.4
45
0.4 0.6
Figure 14: Result of the square experiment in terms of the In1 distance between the network error and the theoretical lower bound in percentage. Each cell corresponds to an experiment with particular correct ratio (c) and majority label ratio (m). Results are averaged over 10 runs.
r223 3. Let r E G acts on y by pr cns sing where y = kT/22. Note that fixing sin cos
a single value of gives Example 4.11; in other words, this experiment has infinitely many orbits where each orbit is similar Example 4.11.
We generate random polynomial function ʃ that not equivariant, i.e., 3(0, a:) s.t. 9 f(0, x) * (g)y.. Then we to fit f using G-invariant network and G-equivariant network. We measure their error compared with the theoretical lower bound given by Theorem and 4.9. is shown in Table 2, both the invariant network and the equivariant network achieve error rate nearly the same as our theoretical bound. The empirical error is slightly higher than the theoretical error due to the neural network fitting error. Please refer to I.4 for more experiment details.
I Experiment Details
This section describes the details of our experiments. All of the experiment is performed using a single Nvidia RTX 2080 Ti graphic card.
L1 Swiss Roll Experiment
In the Swiss Roll Experiment Section 6.1, we use three-layer MI.P for the unconstrained network. For the a-invariant network, we use a network with two DSS [37] layers to implement the z-invariance, each containing two FC layers. We train the networks using the Adam [24] optimizer with learning rate of 10 The batch size is 128. In each run, there are 200 training data, 200 validation data, and 200 test data randomly sampled from the data distribution. The network is trained for a minimal of 1000 epochs and maximum of 10000 epochs, where the training terminated after there is improvement the classification success rate in the validation set for a consecutive of 1000 epochs. We report the test success rate of the epoch model with the highest validation success rate.
1.2 Square Experiment
In the Square Experiment Section H.2, we use a network with two DSS [37] layers implement the horizontal invariance, where each layer contains two FC layers. We train the networks using the Adam [24] optimizer with a learning rate of 10-3 The batch size is 128. each run, there are 1000 training data, 200 validation data, and 200 test data randomly sampled from the data distribution. The network trained for a minimal of 1000 epochs and a maximum 10000 epochs, where the training is terminated after there is improvement in the classification success rate in the validation set for a consecutive of 1000 epochs. We report the test success rate of the epoch model with the highest validation success rate.
I.3 Digit Classification Experiment
In the Digit Classification Experiment in Section 6.2, we use two similar five-layer convolutional networks for the Da-invariant network and the CNN, where the D4-invariant network is implemented using the e2enn package [60]. Both networks have the similar amount of trainable parameters. We train the networks using the Adam [24] optimizer with learning rate of 5 x 10 and weight decay
22

---

Workspace
Workspace
Observation
Observation
Flip
D Equivariant Policy
(x,
(x,
-0
Flip
Figure 15: The robotic experiment setup and the Dı-equivariant policy network.
of 10-5, The batch size is 256. In each run, there are 5000 training data, 1000 validation data, and 1000 test data randomly sampled from the data distribution. The network trained for a minimal of 50 epochs and a maximum of 1000 epochs, where the training is terminated after there is no improvement in the classification success rate in the validation for consecutive of 50 epochs. We report the test success rate of the epoch model with the highest validation success rate.
1.4 Regression Experiment
In the regression experiment, we validate our theoretical error lower bound for invariant and equivari- ant regression (Theorem 4.8, 4.9) by comparing empirical network fitting error and the theoretical
fitting error of function f. Specifically, the function f maps a distance 0 and an index x pair to vector y:
f:Rx R given by (0, +
(25)
where X {arp= 21, F2, x3). The group g E = C4 (e, r2, و3ر acts on (0, by g(0,x) (0, gr) through permuting the index Iι = rroi: 121133 rT2; To rw3. Let & E G acts
vector y by rotation py
cosg sing coss where g /2
We construct function in the following way: for each x E X, choose 1, I + R and define f0, (6). Notice that when 1y2 = Py (g)
G-equivariant. We define (,) (pm (0). (0)) where Ps and Ou are cubic polynomials of i.g. Pa with coefficients b, d will be P> ax + bxx" ex + d. We choose pa and q, with different coefficients each x such that is not equivariant, i.e., 10= * pr (0). For each run, generate function f. sample data 0, and evaluate the data obtaining y. Then we train neural networks using L2 loss till converge. Eventually, sample another set of data evaluate the empirical L2 error well the theoretical L2 error.
1.5 Robotic Experiment
In the robotic manipulation experiment, the state is defined top-down RGBD image of the workspace centered the gripper's position (Figure 15 middle). The action = (x, y, 2, 0, A) defined the change of position (x, 2) and top-down orientation (0) of the gripper, with the
23

---

gripper open width (A), For a D1 = {1,9) group where g represents horizontal flip. the group action on the state space gs defined flipping the image: the group action on the action space ga is defined as flipping the and 0 action and leaving the other action components unchanged, -0, A). We define Dj -yeeuivvrrinnt policy network и 8 - a using e2enn [60], where the output action of π will flip accordingly when the input image is flipped (Figure 15 bottom). We train the network using the Adam [24] optimizer with a learning rate of 10-3 and weight decay of 10 For each run, we train the network for a total of 20k training steps, where we perform evaluation for 100 episodes every 2k training steps. We report the highest success rate of the 10 evaluations as the result of the run.
ga = (z, y. -.
We develop the experimental environments the PyBullet [12] simulator, based on the BullatArm benchmark [57]. In the Stacking (correct equivariance) and Sorting (incorrect equivariance) experi- ment, we gather total of 400 episodes of demonstrations, where 400c of them are Stacking and the rest 400(1 c) are Sorting. In evaluation, the task follows the same distribution, where 100c of the evaluation episodes are Stacking and the rest are Sorting. Notice that the agent can distinguish the Stacking and Sorting tasks because the object colors are different for the two tasks (green and blue for stacking, yellow and red for sorting). the Sorting (extrinsic equivariance) experiment, we also use 400 episodes of demonstrations.
Specifically, in Sorting, the cube and the triangle are initially placed randomly, within a distance of =1.cmm from the horizontal mid-line of the workspace. The objective is to push the triangle at least 9cm toward left and push the cube at least 9cm toward right, while ensuring that both objects remain within the boundaries of workspace. In Stacking, two blocks are randomly initialized on the floor of the workspace. The goal is pick up the triangle and place it on top of the cube. The workspace has size of 30cm x 30cm x 25cm.
24


Reviews:
[{"review_id": "eJQXxvPVMP", "reviewer": "Reviewer_FMJP", "summary": "This work addresses the situation in which data and model equivariance does not match exactly. It extends previous work from Wang et al., by proposing a pointwise version of their definition of correct, incorrect and extrinsic equivariance. The usefulness of these new concepts are demonstrated in three new performance lower-bounds derived from classification and regression problems. A series of examples and experiments are presented to illustrate the new proposed notions, confirm the lower-bounds derived and show that they usually seem tight in practice.", "questions": "See 3 questions in the weaknesses section above.", "limitations": "Yes, the authors mention one limitation of their work, which is the need to know $p$, which is never the case in practice.  ", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "### Originality\nAlthough it builds heavily on previous work from Wang [52], the paper presents novel theoretical concepts and results, as well as new experiments supporting them.\n\n### Clarity\nOverall, the paper is well written and structured.\n\n### Quality\nThe paper presents new theoretical results with their corresponding proofs, which look sound to be (disclaimer: I have some familiarity with group theory, but I am not an expert). It also presents interesting results in a large range of experiments. The latter are all rather small/toy, but are still quite convincing IMO.\n", "weaknesses": "### Clarity\n\nMinor:\n\n- I think there is a mistake in the xlabel of fig 7b. Shouldn't it be \"incorrect - correct\"? Same for figure 7a: should be \"correct - extrinsic\" I guess, since when x=1 you have the highest INV model performance, which should correspond to c=1 and e=0 according to the text.\n- L.363: I think you wanted to reference fig.9, not 8.\n\n### Clarity and Quality\n\n1. I think the definition of $p$ could be clarified. It is introduced in line 92 as the “probability density function of the domain”, so, at first reading, I thought p(x) was the “true” underlying population distribution from which both training and test examples are samples. This seems to be confirmed by line 387 in the conclusion: “our theoretical lower bounds require domain knowledge like the density function over the domain”.  But in section 6, it seems that you define extrinsic equivariance with respect to the actual examples in the training set, which is not the same thing: “F_E corresponds to an extrinsically equivariant class for **this data**” l.272\n\n2. I also have other few questions regarding the example presented in section 6. First of all, does the data from figure 5 represent training or test data? Is the data $S$ the whole support of $p(x)$? What is the true labeling function $f$ in this example?\nThese elements seem important to conclude. The reason I ask is that if $f$ is indeed the “exclusive or” on (x,y) coordinates (i.e. it is indeed C2-invariant) and if the examples from figure 5a are just the training data but the test data can go outside (e.g. their symmetric elements in fig 5b could be the test set), then despite the 0% error rate of the unconstrained linear model on the training data, it would learn an incorrect labeling function and its test performance would be 50%, while the invariant model would still have a 25% error rate, which would be better.\n\n3. If $p$ really denotes the population distribution, I wonder whether we can talk about $p$ independently of $f$ and vice-versa. From your figures 1 and 2, it seems that $f$ can be defined outside of the support of $p(x)$, and I wonder whether this makes sense. For example, if $f$ is the labeling function in a digit classification problem, what should be its output for an image of the digit “9” rotated by 90 degrees? Should it be 9 or 6?\n\n### Originality:\n\nMinor:\n\n4. In the related work, some references to class-specific and instance-specific automatic data augmentation works are missing ([1, 2, 3] for example), while they are strongly related to the idea of pointwise invariance proposed in the paper.\n\n[1] https://arxiv.org/abs/2106.13695\n\n[2] https://arxiv.org/abs/1510.02795\n\n[3] https://arxiv.org/abs/2206.00051", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The authors thank the reviewer for their careful review. Please see our response below.\n\n> I think there is a mistake in the xlabel of fig 7b. \n\nWe apologize for any confusion regarding the `-` in `incorrect - correct` and the other x label in Fig 7. The `-` here is not meant to denote subtraction, rather, it indicates that the distribution transitions linearly from incorrect to correct as we move along the x-axis from left to right. We appreciate your comment and will revise the label to make it more explicit by changing the x label of Fig 7a to ‘correct ratio’, and labeling the x=0 as ‘c=0, e=1’ and labeling x=1 as ‘c=1, e=0’. \n\n> L.363: I think you wanted to reference fig.9, not 8.\n\nWe appreciate your careful reading. Indeed, there is a referencing error. We will correct it in the revised version.\n\n> I think the definition of $p$ could be clarified...\n\nThank you for pointing this out. $p$ is indeed the probability density function of the domain and this description was omitted in Section 6. We consider the probability density $p$ to be uniform for this domain, where the data domain consists only of the 4 samples and $p$ is zero everywhere else. The $C_2$ group action acts along the z-axis and the transformed data contains samples that were not part of the original domain (e.g., $x=(0,0,-1), gx=(0, 0, 1)$). We will add these clarifications to the final revision.\n\n> I also have other few questions regarding the example presented in section 6. First of all, does the data from figure 5 represent training or test data? Is the data $S$ the whole support of $p(x)$? What is the true labeling function $f$ in this example? These elements seem important to conclude. The reason I ask is that if $f$ is indeed the “exclusive or” on (x,y) coordinates (i.e. it is indeed C2-invariant) and if the examples from figure 5a are just the training data but the test data can go outside (e.g. their symmetric elements in fig 5b could be the test set), then despite the 0% error rate of the unconstrained linear model on the training data, it would learn an incorrect labeling function and its test performance would be 50%, while the invariant model would still have a 25% error rate, which would be better.\n\nIn this example, we do not differentiate between training and test data and focus on only the learnability of a non-equivariant/invariant function class for a specific distribution and not on its generalization. The domain is discrete where Figure 5a shows the only four possible values of x with probabilities p(x) = 1/4. As we consider the model’s capacity to fit arbitrary labelings (since we use Rademacher complexity), there is no ground truth labeling function and Figure 5a shows one instance of an assignment of labels. We consider the training set S to be all 4 data points. Section 6 serves to show that for a linear class of models, an extrinsically-equivariant/invariant model has a non-zero error rate for this data. The example you compute is correct if the test data contained the 8 points shown in Figure 5b, but we do not consider this scenario as it is not an example of extrinsic equivariance. The test and training data is only the 4 points shown in Figure 5a. We will add these clarifications to the text.\n\n> If $p$ really denotes the population distribution, I wonder whether we can talk about $p$ independently of $f$ and vice-versa. From your figures 1 and 2, it seems that $f$ can be defined outside of the support of $p(x)$, and I wonder whether this makes sense. For example, if $f$ is the labeling function in a digit classification problem, what should be its output for an image of the digit “9” rotated by 90 degrees? Should it be 9 or 6?\n\nThank you for raising this question. In our analysis, we only consider the behavior of $f$ within the support of $p$ because the output of $f$ outside the support of $p$ will not change our bounds. Note that in Equations 2 and 3, the equations are weighted by $p$, so for x outside the support of $p$, the value of $f$ does not matter. In Figures 1 and 2, we draw $f$ outside of the support of $p$ to make it continuous for ease of understanding. If they are confusing, we are happy to iterate on them to remove the part where $p(x)=0$.\n\nIn practice, there can be circumstances where defining $f$ outside the support of $p$ is meaningful. For example, $f$ can be expanded outside the support of $p$ in scenarios like a random crop data augmentation, where the output of $f$ will not change after a random crop, even though the cropped image is out of distribution. In the reviewer’s example regarding digits 6 and 9, if the rotated 6 is outside the support of $p$ (like in MNIST where the rotated 6 and 9 are different due to handwriting), $f$’s output would be undefined.\n\n> In the related work, some references to class-specific and instance-specific automatic data augmentation works are missing ([1, 2, 3] for example), while they are strongly related to the idea of pointwise invariance proposed in the paper  \n[1] https://arxiv.org/abs/2106.13695  \n[2] https://arxiv.org/abs/1510.02795  \n[3] https://arxiv.org/abs/2206.00051\n\nThank you for pointing out those references! Yes, those related works are strongly related to our work. First, the data augmentation methods can be viewed as learning pointwise extrinsic equivariance with respect to transformations defined by the augmentation function. Second, the instance and class-specific augmentations can also be viewed as applying data augmentation where pointwise correct or extrinsic invariance exist, while avoiding incorrect invariance. Moreover, while our current work is focused on constrained model classes, our theory can also potentially be applied to analyze these data augmentation methods. We will definitely add them to the related work section. \n"}, {"review_id": "k8FDLoEjWO", "reviewer": "Reviewer_1dq7", "summary": "The paper presents an extension of a lower bound on error in finite labeled classification introduced in \"The Surprising Effectiveness of Equivariant Models in Domains with Latent Symmetry\" and investigates lower bounds on error for G-equivariant and G-invariant regression, offering valuable insights into the performance of symmetry-preserving models in regression tasks.", "questions": "1. \nCan you elaborate on the third experiment? Which state is flipped? (line 532, page 9)\n\n2.\nDid you mean \"orbits\" instead of conjugates in the definition of the fundamental domain? (line 158, page 4)", "limitations": "The author did address the limitations of reflection symmetries on the expressivity of models in Chapter 6 (line 254, page 7), and the requirement of knowledge about the density function (line 386, page 9)\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This extension holds significant practical importance, particularly when considering real-world natural symmetry groups such as $G = SO(3)$, which go beyond the scope of the original paper. Moreover, the authors have done an excellent job in presenting the material in an accessible manner, making the paper easy to read and well-structured.", "weaknesses": "1.\nThe paper's focus on pointwise definitions rather than emphasizing their implications may obscure the novelty of the results, potentially hindering a clear understanding of the significance of their contributions.\n\n2.\nWhile the experiments conducted in the paper offer valuable insights, their omission of infinite groups may limit the demonstration of the full strength of the new lower bound. Additionally, the observations suggesting equivariant models' potential ineffectiveness in classical learning tasks raise valid questions about their relevance and applicability.\n\n3.\nAlthough the paper draws upon certain notions from \"The Surprising Effectiveness of Equivariant Models in Domains with Latent Symmetry,\" it does so only partially. This partial utilization of concepts, coupled with the use of different evaluation metrics such as \"accuracy\" in the original paper and \"error\" in this work, may create challenges when attempting to directly compare the results between the two studies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The authors appreciate the reviewer’s insightful comments and questions. Please see our response below.\n\n> The paper's focus on pointwise definitions rather than emphasizing their implications may obscure the novelty of the results, potentially hindering a clear understanding of the significance of their contributions.\n\nWe appreciate your feedback. The pointwise definitions are a generalization of the prior work [52]. They help us analyze the problem of incorrect/extrinsic equivariance more completely and can be used to calculate the lower bound, as is shown in Proposition 5.4. However, the major contribution of our work is the bounds in Section 5 instead of the pointwise definitions. We plan to merge Section 4 into Section 3 and iterate on our writing to make the contributions clearer.\n\n> While the experiments conducted in the paper offer valuable insights, their omission of infinite groups may limit the demonstration of the full strength of the new lower bound. \n\nThanks for the comment. While our theoretical findings are not restricted to finite groups, we opted to use finite groups in our examples and experiments for clarity and ease of understanding. \n\n> Additionally, the observations suggesting equivariant models' potential ineffectiveness in classical learning tasks raise valid questions about their relevance and applicability.\n\nEquivariant models have been shown to be effective in many learning tasks. However, given the constrained nature of equivariant models, it is possible that they will lead to inevitable errors in some cases. Our study explores these potential scenarios of ineffectiveness, which we believe can provide valuable insights for model selection and analysis. \n\n> Although the paper draws upon certain notions from \"The Surprising Effectiveness of Equivariant Models in Domains with Latent Symmetry,\" it does so only partially. This partial utilization of concepts, coupled with the use of different evaluation metrics such as \"accuracy\" in the original paper and \"error\" in this work, may create challenges when attempting to directly compare the results between the two studies.\n\nOur work builds upon the prior work “The Surprising Effectiveness of Equivariant Models in Domains with Latent Symmetry” to extend a theory that the prior work only partially addresses. Consequently, some of their definitions and framework are too limited and make too many assumptions. Removing these assumptions and generalizing that work is a key part of our  goal. For example, the prior work limits their analysis in classification tasks, so using “accuracy” makes sense in their context. However, our work aims to calculate a more accurate bound for both classification and regression, thus “error” would be a better choice.\n\n> Can you elaborate on the third experiment? Which state is flipped? (line 532, page 9)\n\nThanks for asking this question. The state is a top-down RGBD image of the workspace, and by flipping the state, we mean that the image is flipped horizontally. Similarly, flipping the action means that the (x, y) component of the robot action is flipped horizontally. Due to the page limit, those details were put in Appendix K.5 instead of the main paper.\n\n> Did you mean \"orbits\" instead of conjugates in the definition of the fundamental domain? (line 158, page 4)\n\nWe appreciate the question. However, we indeed meant \"conjugates\" not \"orbits\". An orbit of $x\\in X$ is defined as the set $\\\\{ gx| g\\in G\\\\}$, whereas a conjugate $gF$ is defined as $gF=\\\\{gx | x\\in F\\\\}$ (please see https://mathworld.wolfram.com/FundamentalDomain.html). We understand that this could be a point of confusion, and we will ensure to clarify the definition of a conjugate in the revision. "}, {"review_id": "nqsPqbqxw8", "reviewer": "Reviewer_racn", "summary": "The paper analyzes error bounds for models constrained to satisfy symmetries that only partially agree with the ground truth functions. The main suggestion is to generalize previous definitions, to the point level, allowing to derive a lower bound on the model error with respect to the volume of the portion of the domain where symmetries are mismatched. The theoretical analysis is validated on some toy examples and some (relatively) small-scale datasets", "questions": "No further questions.", "limitations": "Yes", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed analysis including the definitions, propositions, and theorems seems to be simple.\n\nThe paper is well-written and easy to follow. I appreciate the illustrations and toy examples provided. \n\nThe experiments seem to support the theoretical analysis. Where mismatches occur, reasonable explanations are provided.\n", "weaknesses": "Missing discussion\n\nI feel the text should elaborate more on the assumptions taken in the analysis of error bound. For example, it is assumed that the model assigns labels by taking a majority vote in an orbit. How reasonable is this assumption in relation to existing equivariant models? \n\nIn addition, it would be interesting to incorporate a model that approximates the majority vote by sampling for one of the experiments.  E.g., does it improve the digit classification model?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The authors thank the reviewer for their thoughtful review. Please see our response below.\n\n> I feel the text should elaborate more on the assumptions taken in the analysis of error bound. For example, it is assumed that the model assigns labels by taking a majority vote in an orbit. How reasonable is this assumption in relation to existing equivariant models? \n\nNote that label assignment by majority vote is *not* an assumption of Theorem 5.3, which applies to any invariant $h$. We will clarify the hypothesis in the theorem statement. The $h^*$ which uses majority voting is part of the proof for Theorem 5.3. Intuitively, we compute a lower bound on $err(h)$ for any invariant $h$ by comparing it to the error of the best case invariant hypothesis $err(h^*)$. If the model were not to follow this majority voting scheme, it would make more mistakes in an orbit, resulting in a higher error (as illustrated in the Equation at the end of page 4). \n\n> In addition, it would be interesting to incorporate a model that approximates the majority vote by sampling for one of the experiments. E.g., does it improve the digit classification model?\n\nThank you for suggesting this experiment. We interpret the proposed approach as follows: instead of learning an invariant model for the digit classification task, we will have a sample-based model. To predict the label of $x$, this model will access $f(gx)$ for all $g\\in G$ and calculate the majority vote as the output. This is a very interesting approach, but we think it might not be practical in our experiment because it will need access to an oracle ground truth function $f(x)$ during evaluation. Please let us know if we didn’t interpret your suggestion correctly.\n"}, {"review_id": "4LOuD707GP", "reviewer": "Reviewer_bWzk", "summary": "An obvious limitation of equivariant networks is the assumption that the symmetry they hard-code matches the symmetry of the underlying ground truth function exactly. What happens when the symmetry is only partially present in the domain affecting a symmetry mismatch between the ground-truth function and the neural network? This paper presents some systematic analysis of this situation building on prior work by Wang et al. 2023.\n\nWang et al. classified models as having \"correct\" equivariance (the underlying function provided by nature matches the symmetry of the equivariant model), \"incorrect\" equivariance (there is a disagreement between the underlying true function and that encoded by the equivariant network), and \"extrinsic\" equivariance (model symmetry transforms in-distribution data to out-of-distribution data). This paper defines these notions pointwise which allows one to generalize to a continuum of equivariance types when some tasks may have different proportions of these three situations (and does not exclusively conform to only one of them). Lower bounds are provided on the model error from incorrect model symmetry (while improving some earlier results) for classification. Morally similar lower bounds are provided for the regression case in terms of the variance of the function to be modelled over the orbit of the group under consideration. It is also shown that extrinsic equivariance can be harmful (which doesn't contradict but can still be contrasted with earlier experimental results in the literature).\n\nThe theoretical results are provided in section 5 (after first introducing the notions of correct, incorrect and equivariance of Wang et al, and then stating their pointwise generalizations which open the path to interpolate between them -- a central aim of the paper). The lower bound for classification is intuitive and basically says that it is equal to the integral of the total dissent over the fundamental domain. The total dissent measures how many elements in the orbit of G have a different label than the majority label. Two examples are given for binary and multiclass cases to build intuition (also illustrated in figures 3 and 4). Next, the case of invariant regression is treated. The result is similar in spirit to the result from earlier -- the error is bounded by an integral over the fundamental domain of p(Gx) times the variance of the function on the orbit Gx (so instead of the dissent, we look at the variance). A slightly more careful consideration permits an easy generalization for equivariant regression. A simple theoretical argument is then provided to show that extrinsic can be harmful for generalization, also raising the question that it remains open to understanding its effect on generalization.", "questions": "No", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper considers an important problem that has only recently started getting treated in the literature. It is clear that equivariant networks make a strong assumption about the symmetry of the ground truth function. However, it is not clear how much a mismatch matters. This paper provides some approximation lower bounds due to the mismatch for some different settings. \n-  Being able to interpolate between the three types can definitely help improve model generalization and permit more flexible models.\n- While many of the points raised in the paper seem somewhat obvious at times, and computing the lower bound might not be possible in practice, the presented work can still give some guidance on model selection.", "weaknesses": "- Somewhat fortuitously, I was a reviewer for this paper for an earlier iteration. Whatever weaknesses that I had raised at the point seem to have been adequately addressed by the authors (in fact they were right then). I also notice that the other weaknesses bought up by other reviewers at the time (mostly in terms of experiments, and comparison to the work of Wang) have also mostly been incorporated. While I can nitpick, I have no hesitation in simply voting for acceptance. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The authors thank the review for their careful review. We are glad that the reviewer acknowledges that we have addressed the problems from the earlier iteration. If you have any other questions regarding our paper, please don’t hesitate to let us know and we are more than happy to discuss them."}, {"review_id": "LtiReb7xPp", "reviewer": "Reviewer_yJf8", "summary": "This paper provides a general theory of correct, incorrect, and extrinsic equivariance of functions, mainly extending the framework of Wang et al., 2023 to more general case of pointwise equivariance of functions defined for pairs of group element and input data. The theory mainly concerns deriving lower bound of errors for classification and (invariant and equivariant) regression tasks. The authors further identify cases where extrinsic equivariance can be harmful for performance opposed to the empirical observations of Wang et al., 2023. The authors provide a range of experiments mainly on empirically verifying the derived lower bound of errors, and also demonstrating the cases where certain extrinsic equivariances can be harmful.\n\nWang et al., The surprising effectiveness of equivariant models in domains with latent symmetry (2023)", "questions": "I have no particular questions, but would like to hear the opinions of the authors on the aforementioned weaknesses.", "limitations": "The authors have identified the limitation of the work in Section 8.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. The theory proposed in the paper is indeed quite general as it considers pointwise equivariance, and covers a large range of partial symmetries. The fact that it addresses a wide range of tasks (classification, invariant regression, equivariant regression) also strengthens the generality of the paper.\n\nS2. The paper is overall well written, with intuitive illustrations on pointwise equivariance and experimental setups, as well as results.\n\nS3. The experimental results support the main claims of the paper on error bounds and harmful cases of extrinsic equivariances.", "weaknesses": "W1. A weakness of this work is that, while certain cases are presented where extrinsic equivariances can be harmful, it offers little principled understanding or general theory of in which specific cases extrinsic equivariance is harmful or beneficial.\n\nW2. While the presented theory on pointwise equivariance covers a wide range of approximate or misspecified symmetries, I am not sure if it is immediately useful, since in many applications involving approximately or misspecified equivariant neural networks, we would not be able to exactly know the extent of correct, incorrect, and extrinsic pointwise equivariance of the model (unlike the synthetic experimental setups considered in the paper). This could be a weakness of the proposed theoretical framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The authors thank the reviewer for their helpful comments. Please see our response below.\n\n> W1. A weakness of this work is that, while certain cases are presented where extrinsic equivariances can be harmful, it offers little principled understanding or general theory of in which specific cases extrinsic equivariance is harmful or beneficial.\n\nYou are right, in the context of our paper, we only show the possibility of extrinsic equivariance being harmful, instead of providing an understanding of when it would be harmful or helpful (in part, this is because the prior work [52] shows helpful extrinsic equivariance and we are trying to do a counterpart). This is a limitation of our work and a very important future direction. We will address this in the limitation section.\n\n> W2. While the presented theory on pointwise equivariance covers a wide range of approximate or misspecified symmetries, I am not sure if it is immediately useful, since in many applications involving approximately or misspecified equivariant neural networks, we would not be able to exactly know the extent of correct, incorrect, and extrinsic pointwise equivariance of the model (unlike the synthetic experimental setups considered in the paper). This could be a weakness of the proposed theoretical framework.\n\nWe agree that in many real-world applications, we might not exactly know the extent of correct, incorrect, and extrinsic pointwise equivariance of the model and it might be hard to compute the lower bound. However, our analysis also provides a theory that can guide the model selection process in such a case. Equivariant models are usually selected on the basis of prior reasoning about properties of the ground truth function. We believe our theory provides intuition which can be part of this prior reasoning process and can be used to understand and analyze model performance in an iterative model development cycle. Moreover, in some real-world applications, we can indeed analyze the extent of different equivariance types. For example, in robotic manipulation, we might know a priori how the objects are distributed on a tabletop, so we can analyze how rotations of the entire workspace change the distribution. \n"}, {"review_id": "NiDJLqrMKa", "reviewer": "Reviewer_c3pV", "summary": "The authors analyze how equivariant models behave under mismatches between the symmetries of the model and data distribution.  They advance three notions, correct, extrinsic, and incorrect pointwise equivariance.  They then propose bounds on the error for equivariant neural networks that are sensitive to the data distribution and give an example where having an equivariant model can hurt learning performance.\n\n(Edit, 8/13: in light of comments by the Authors, I have changed my score to a 5)", "questions": "I’m having a little trouble parsing the notion of Fundamental domain in Definition 5.1.  What happens if the orbit of the passes over X multiple times.  For instance, consider the real line, where the group is S_8 and the group action is the antisymmetric representation (reflect the line on odd permutations)?", "limitations": "The limitations of the work have been appropriately addressed.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Theoretically, the bounds present are more general than those proposed in Wang et al. 52.  The examples clearly demonstrate the underlying mathematical principles involved.  Moreover the error bounds presented, while not unsurprising to practitioners more versed in group theory, seem mathematically solid.  On the whole, the task of attempting to precisely characterize how the data distribution interacts with symmetries imposed on the model is a worthy one.  ", "weaknesses": "While the paper is solid, I think certain difficulties in presentation prevent it from fulfilling its potential.  Most immediately, I think there are some issues in the presentation in the concepts of pointwise correct / incorrect / extrinsic equivariance.  First, equivariance is a property of maps rather than spaces: saying a point has correct / incorrect / extrinsic equivariance, or even equivariance of any kind, isn’t precise.  It would be better to say something the lines of “h has correct / incorrect / extrinsic equivariance with respect to f and p at x”, particularly since the notion of correctness here is dependent on the model: this has the advantage of being a more close parallel with Defs 3.1-3.3. Second (forgive me if I missed it), but at no point is the notion of pointwise equivariance being dependent on specific group elements used in the rest of the paper.  It is not clear what is gained by this added generality.\n\nContinuing on to Section 5, it is not clear to me to what extent the notions advanced in section 4 are necessary for stating these bounds: if I understand correctly the bounds presented are for general probability distributions and are independent of the definitions proposed in 4.1-4.3.\n\nSection 6 seems a bit trivial for me: it seems obvious to me that if you can’t represent a decision boundary due to a constraint in the model you might end up having a bad day.  Indeed, the specific example shown here seems to follow straightforwardly from https://doi.org/10.48550/arXiv.2110.07472.  I think this section should be moved up and be a motivating example for the bounds in Section 5.  \n\nI feel like the meat of the paper is the error bounds, more specifically Theorem 5.8.  I’m not convinced the concepts in Definitions 4.1-4.3 contribute much towards the construction of this bound.  I would recommend a rewrite of the paper that focuses more on the bounds, their implications and applications, and maybe expand further.\n\nFinally, (and this might just be me being a pedant), but I’m not sure the title really is reflective of the content of the paper.  It is not clear to me what the general theory is that is being advanced.  Typically a “general theory” is a precise characterization of a class of mathematical objects.  If this is intended to refer to the decomposition into correct / incorrect / extrinsic equivariance, then this characterization seems to reduce to the fact that that equalities can be true, not true, or involve the empty set.  I don’t think any insight is gained innately from this characterization, rather than this being a useful language to describe further results in the text.  The error bounds, while good contributions, are specific to certain choices of loss function.  I mention this not to diminish their contributions, merely to observe that they are not “general” in a mathematical sense.\n\n\n(edited for some spelling mistakes, although I'm sure more of them abound...)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The Authors thank the reviewer for their insightful review. Please see our response below.\n\n> equivariance is a property of maps rather than spaces...\n\nThis is a good point, we definitely agree with the reviewer that ‘equivariance’ is a property of maps rather than spaces. In the revision, we will revise the definitions based on the reviewer’s suggestions to make them more precise.\n\n> at no point is the notion of pointwise equivariance used here being dependent on specific group elements used in the rest of the paper...  \n\n> it is not clear to me to what extent the notions advanced in section 4 are necessary for stating these bounds...\n\nThe pointwise equivariance is actually used in Proposition 5.4 as a way of calculating the error bound for the classification task. Later on, we also used it in Example 5.5 and the experiment in Section 7.1. The pointwise definitions are necessary for understanding our lower bounds because, intuitively, the bounds calculate the error of the model where pointwise incorrect equivariance exists.\n\n> Section 6 seems a bit trivial...\n\nWe agree that the discussion of Section 6 is not overly complicated. However, we believe it is still important to complete our theory, as extrinsic equivariance can be another source of failure for an equivariant model beyond the incorrect equivariance discussed in Section 5. Moreover, while incorrect equivariance is often considered in the prior works, the effect of extrinsic equivariance is under explored. The prior work by Wang et al. [52] showed that extrinsic equivariance is helpful in all of their experiments. Just by reading [52], one might think that extrinsic equivariance is always helpful, but it is not true. Section 6 provides a discussion about the scenarios when extrinsic equivariance can be harmful.\n\n> Indeed, the specific example shown here seems to follow straightforwardly from https://doi.org/10.48550/arXiv.2110.07472. \n\nAlthough the example In Figure 1 of https://doi.org/10.48550/arXiv.2110.07472 [A] is similar to our example in Section 6, it does not directly follow. Figure 1 in [A] demonstrates that there exists a subspace that is fixed by the group action and that it affects the perceptron capacity. While they consider separating invariant orbits of data samples, our example shows an extrinsically equivariant scenario where the C2 group action produces samples outside of the data domain. For example, the group action transforms $x=(0,0,-1)$ to $gx=(0, 0, 1)$, which is outside of the set of 4 points in Figure 5a and so the symmetry relates in-distribution samples to points that are out-of-distribution. We will clarify these differences in the main text.\n\n>I think this section should be moved up and be a motivating example for the bounds in Section 5. \n\nUnfortunately, we respectfully disagree that Section 6 can be moved up to motivate Section 5, as they discuss the error caused by two different types of equivariance: Section 5 discusses incorrect equivariance and Section 6 discusses harmful extrinsic equivariance. Although both of them can cause the model to have an unsatisfactory performance, the underlying reasons are different. Specifically, incorrect equivariance leads to an inevitable error in the model due to the equivariant constraint, and we can calculate the theoretical lower bound of such an error. On the other hand, harmful extrinsic equivariance increases the complexity of the task, but unlike incorrect equivariance, it can be solved by increasing model capacity.\n\n>I feel like the meat of the paper are the error bounds...\n\nThis is a good suggestion, thanks for pointing this out. We agree that the most important contribution of the paper is Section 5, rather than Section 4. In the revision, we propose to make the following changes to our paper:\n1. We will rework the story to focus more on the contribution of the paper: the cases where equivariant models might underperform due to approximation errors and harmful extrinsic equivariance.\n2. We plan to merge Section 3 and Section 4 into a combined background and preliminaries section. \n\n>I’m not sure the title really is reflective of the content of the paper...\n\nThank you for the comment. Our paper builds heavily upon [52] where the authors defined the terms of correct, incorrect, and extrinsic equivariance. However, their discussion is not complete as they assume that the data density over the domain is group invariant and that the group is a finite group. Our work generalizes theirs by removing those simplifying assumptions, discussing the lower bounds for regression (rather than just classification as in the prior work), and discussing the possibility of extrinsic equivariance being harmful. Thus, we named our paper “A General Theory”. Nevertheless, we don’t mean to misuse the term “general” in a mathematical sense, and we are happy to remove the word “general” in the title of our paper.\n\n>I’m having a little trouble parsing the notion of Fundamental domain...\n\nGenerally, if the orbit passes over $X$ multiple times, the stabilizer will be non-empty, and the factor $\\alpha(x, g)$ in Equation 3 will account for the over-counted conjugates. In the example, the group $S_8$ acts on $\\mathbb{R}$ s.t. $s_i \\cdot x = -1^{i}x$ for $x\\in \\mathbb{R}, s_i\\in S_8$, the fundamental domain $F$ would be either $\\\\{x>=0 | x\\in\\mathbb{R}\\\\}$ or $\\\\{x<=0 | x\\in\\mathbb{R}\\\\}$. Notice that this does not violate our assumption in line 161 because $\\cup_{g_1F \\neq g_2F} (g_1F \\cap g_2F) = \\\\{0\\\\}$ has 0 measure under $\\nu$. Since $S_8$ is a finite group, Equation 3 becomes $k(Gx) =\\min_{y\\in Y} \\sum_{g\\in S_8}p(gx) \\mathbb{1} (f(gx) \\neq y) \\alpha(x, g) dg$, where $\\alpha(x, g)=\\frac{2}{8}$ will account for the over-counted conjugates. \n"}]

Author rebuttal:

Meta-review: Given a group action and a distribution on the input space, a function may or may not be equivariant when restricted to the support of the distribution (assuming in addition an action on the output space). Wang et al. [52], considering mismatches between model symmetries and data distributions, also studied a third case in which the support is not invariant to the group action. The paper considers a generalization of Wang et al. [52] to pointwise equivariance, allowing situations in which the three cases considered hold in different parts of the domain.

The reviewers appreciated the generality as well as the practical relevance of the relaxed notion of equivariance considered in the paper, so as the series of experiments provided by the authors to demonstrate the validity of lower bounds so derived. After the rebuttal phase, an attempt was made to come to a consensus based on the assumption that as section 5 offers limited technical contribution, evaluation mainly hinges on possible conceptual contributions, i.e., a natural refinement of the framework introduced by [52]. Despite the discussions with the authors, those who initially gave borderline feedback largely maintained their assessment, believing that the conceptual framework, though well-presented, may either not be sufficiently provoking or that its applicability may be severely restricted by the requirement to have access to the underlying data distribution (or a proxy thereof whose quality has not been properly discussed\established). Those who gave high scores raised, by and large, similar concerns.

Eventually, following the considerable discussion the paper has generated, the reviewers coincided on: a reasonable paper to accept if there is room.
Decision: Accept (poster)