Title: A General Theory of Correct, Incorrect, and Extrinsic Equivariance
Venue: NeurIPS 2023
Accepted: yes
Keywords: Equivariance

Deep Learning

Error Bound

Symmetry
Abstract: Although equivariant machine learning has proven effective at many tasks, success depends heavily on the assumption that the ground truth function is symmetric over the entire domain matching the symmetry in an equivariant neural network. A missing piece in the equivariant learning literature is the analysis of equivariant networks when symmetry exists only partially in the domain. In this work, we present a general theory for such a situation. We propose pointwise definitions of correct, incorrect, and extrinsic equivariance, which allow us to quantify continuously the degree of each type of equivariance a function displays. We then study the impact of various degrees of incorrect or extrinsic symmetry on model error. We prove error lower bounds for invariant or equivariant networks in classification or regression settings with partially incorrect symmetry. We also analyze the potentially harmful effects of extrinsic equivariance. Experiments validate these results in three different environments.

A General Theory of Correct, Incorrect, and Extrinsic Equivariance

Abstract

Although equivariant machine learning has proven effective at many tasks, success depends heavily on the assumption that the ground truth function is symmetric over the entire domain matching the symmetry in an equivariant neural network. A missing piece in the equivariant learning literature is the analysis of equivariant networks when symmetry exists only partially in the domain. In this work, we present a general theory for such a situation. We propose pointwise definitions of correct, incorrect, and extrinsic equivariance, which allow us to quantify continuously the degree of each type of equivariance a function displays. We then study the impact of various degrees of incorrect or extrinsic symmetry on model error. We prove error lower bounds for invariant or equivariant networks in classification or regression settings with partially incorrect symmetry. We also analyze the potentially harmful effects of extrinsic equivariance. Experiments validate these results in three different environments.

1 Introduction

Equivariant neural networks [9, 10] have proven to be an effective way to improve generalization and sample efficiency in many machine learning tasks. This is accomplished by encoding task-level symmetry into the structure of the network architecture so that the model does not need to explicitly learn the symmetry from the data. However, encoding a fixed type of symmetry like this can be limiting when the model symmetry does not exactly match the symmetry of the underlying function being modeled, i.e., when there is a symmetry mismatch. For example, consider the digit image classification task. Is it helpful to model this problem using

(a) Correct

(b) Incorrect

(c) Extrinsic

Figure 1: An example of correct, incorrect, and extrinsic equivariance. The ground truth function f ( x ) is shown in black and its probability density function p ( x ) is shown in orange. If we model f ( x ) using a G -invariant network where G is a reflection group that negates x , different f ( x ) and p ( x ) will lead to correct, incorrect, and extrinsic equivariance. See Section 3 for details.

a model that is invariant to 180-degree rotation of the image? For some digits, the label is invariant (e.g., 0 and 8 ). However, for other digits, the label changes under rotation (e.g., 6 and 9 ), suggesting that a rotationally symmetric model would be inappropriate here. However, recent work [58] suggests that this is not necessarily the case - symmetric models are sometimes helpful even when a symmetry mismatch exists between the problem and model. This raises the question - do the advantages obtained by using a symmetric model outweigh the errors introduced by the symmetry mismatch?

This paper makes four main contributions towards this problem. First, this paper extends the definitions for types of model symmetry with respect to true symmetry introduced in Wang et al. [58]. They classify models as having correct , incorrect , or extrinsic equivariance (see Figure 1), where correct means the ground truth function has the same symmetry as the equivariant model, incorrect means the ground truth function disagrees with the symmetry in the model, and extrinsic means the model symmetry transforms in-distribution data to out-of-distribution data. We generalize this system into a continuum of equivariance types to reflect the fact that a single task may have different proportions of correct, incorrect, and extrinsic symmetry across its domain. For example, in the digit classification task, 0 has correct equivariance, 6 has incorrect equivariance, and 4 has extrinsic equivariance.

Our second contribution is to introduce an analytical lower bound on model error in classification tasks resulting from incorrect model symmetry. This result can help guide model selection by quantifying error resulting from incorrect equivariance constraints. Our result generalizes that of Wang et al. [58] by removing the simplifying assumption that data density over the domain is group invariant. We prove the minimum error of an invariant classifier can be realized by assigning all data points in the same group orbit the label with the majority of the data density (Theorem 4.3).

Our third contribution is to develop new lower bounds on the L 2 error for regression tasks in terms of the variance of the function to be modeled over the orbit of the symmetry group. Like our classification bound, this bound can assist in model selection in situations with symmetry mismatch.

Fourth, in contrast to Wang et al. [58] who show benefits of extrinsic equivariance, we theoretically demonstrate its potential harm. We perform experiments documenting the error rate across the correct-extrinsic continuum. Finally, we perform empirical studies illustrating the ideas of the paper and showing that the lower bounds obtained in our analysis appear tight in practice. This suggests our analysis can assist practitioners select symmetry groups appropriate for a given problem setting. Our code is available at https://github.com/pointW/ext_theory .

2 Related Work

Equivariant Learning. Originally used for exploiting symmetry in image domains [9, 10], equivariant learning has been very successful in various tasks including molecular dynamics [2, 4], particle physics [6], fluid dynamics [59], trajectory prediction [53], pose estimation [31, 26, 32], shape completion [7], robotics [48, 65, 21, 55, 49, 41, 23, 22, 45] and reinforcement learning [52, 54, 56, 39, 63]. However, most prior work assumes that the symmetry of the ground truth function is perfectly known and matches the model symmetry. Wang et al. [58] go further and define correct, incorrect, and extrinsic equivariance to classify the relationship between model symmetry and domain symmetry. However, they do not discuss the possible combinations of the three categories, and limit their theory to a compact group and invariant classification. Our work extends [58] and allows for a continuum of equivariance types and analyzes error bounds in a more general setup.

Symmetric Representation Learning. Various works have proposed learning symmetric representations, using transforming autoencoders [20], restricted Boltzmann machines [50], and equivariant descriptors [47]. In particular, [30] shows that convolutional neural networks implicitly learn representations that are equivariant to rotations, flips, and translations, suggesting that symmetric representations are important inductive biases. Other works have considered learning symmetryaware features using disentanglement [43], projection mapping [25], equivariance constraints [35], separation into invariant and equivariant parts [61] or subgroups [34]. Park et al. [42] propose learning a symmetric encoder that maps to equivariant features and Dangovski et al. [13] learn features that are sensitive and insensitive to different group representations. Other works assume no prior knowledge of symmetry and learn it from data [3, 64, 14, 40]. In particular, Moskalev et al. [40] estimate the difference between the true latent symmetry and the learned symmetry. Similarly, our work considers a gap between the true symmetry and model symmetry and theoretically analyze its effects on error.

Theory of Equivariant Learning. There are several lines of work on the theory of equivariant learning. Kondor and Trivedi [27] prove that convolutions are sufficient and necessary for equivariance of scalar fields on compact groups, later generalized to the steerable case by Cohen et al. [11]. Certain equivariant networks have been proved to be universal in that such networks can approximate any G -equivariant function [36, 62]. Another line of work has considered equivariant networks in terms of generalization error. Abu-Mostafa [1] show that an invariant model has a VC dimension less than

or equal to that of a non-equivariant model. Other works studied the generalization error of invariant classifiers by decomposing the input space [51, 46]. Elesedy and Zaidi [17] quantify a generalization benefit for equivariant linear models using the notion of symmetric and anti-symmetric spaces. A PAC Bayes approach was used for generalization bounds of equivariant models [5, 33]. Our work is complimentary to these and quantifies approximation error for equivariant model classes.

Data Augmentation. Some methods use data augmentation [29, 28] to encourage the network to learn invariance with respect to transformations defined by the augmentation function [8]. Recent works have explored class-specific [19, 44] and instance-specific [38] data augmentation methods to further boost training by avoiding the potential error caused by a uniform augmentation function. Those methods can be viewed as applying data augmentation where pointwise correct or extrinsic invariance exist, while avoiding incorrect invariance.

3 Preliminaries

Problem Statement. Consider a function f : X → Y . Let p : X → R be the probability density function of the domain X . We assume that there is no distribution shift during testing, i.e., p is always the underlying distribution during training and testing. The goal for a model class { h : X → Y } is to fit the function f by minimizing an error function err( h ) . We assume the model class { h } is arbitrarily expressive except that it is constrained to be equivariant with respect to a group G . Let ✶ be an indicator function that equals to 1 if the condition is satisfied and 0 otherwise. In classification, err( h ) is the classification error rate; for regression tasks, the error function is a L 2 norm function,

̸

err cls ( h ) = E x ∼ p [ ✶ ( f ( x ) = h ( x ))] , err reg ( h ) = E x ∼ p [ || h ( x ) -f ( x ) || 2 2 ] . (1) Equivariant Function. A function f : X → Y is equivariant with respect to a symmetry group G if it commutes with the group transformation g ∈ G , f ( gx ) = gf ( x ) , where g acts on x ∈ X through the representation ρ X ( g ) ; g acts on y ∈ Y through the representation ρ Y ( g ) .

3.1 Correct, Incorrect, and Extrinsic Equivariance.

Consider a model h which is equivariant with respect to a group G . Since real-world data rarely exactly conforms to model assumptions, in practice there may often be a gap between the symmetry of the model and the ground truth function. Wang et al. [58] propose a three-way classification which describes the relationship between the symmetry of f and the symmetry of h . In this system, h has correct equivariance, incorrect equivariance, or extrinsic equivariance with respect to f .

Definition 3.1 (Correct Equivariance) . For all x ∈ X,g ∈ G where p ( x ) > 0 , if p ( gx ) > 0 and f ( gx ) = gf ( x ) , h has correct equivariance with respect to f .

̸

Definition 3.2 (Incorrect Equivariance) . If there exist x ∈ X,g ∈ G such that p ( x ) > 0 , p ( gx > 0) , but f ( gx ) = gf ( x ) , h has incorrect equivariance with respect to f .

Definition 3.3 (Extrinsic Equivariance) . For all x ∈ X,g ∈ G where p ( x ) > 0 , if p ( gx ) = 0 , h has extrinsic equivariance with respect to f .

Example 3.4. Consider a binary classification task where X = R and Y = { 0 , 1 } . If the model h is invariant to a reflection group G where the group element g ∈ G acts on x ∈ X by gx = -x , Figure 1 shows examples when correct, incorrect, or extrinsic equivariance is satisfied.

3.2 Pointwise Equivariance Type.

Although Definitions 3.1- 3.3 are self-contained, they do not consider the mixture of different equivariance types in a single function. In other words, an equivariant model can have correct, incorrect, and extrinsic equivariance in different subsets of the domain. To overcome this issue, we define pointwise correct, incorrect, and extrinsic equivariance, which is a generalization of the prior work.

̸

̸

Definition 3.5 (Pointwise Correct Equivariance) . For g ∈ G and x ∈ X where p ( x ) = 0 , if p ( gx ) = 0 and f ( gx ) = gf ( x ) , h has correct equivariance with respect to f at x under transformation g .

̸

Definition 3.6 (Pointwise Incorrect Equivariance) . For g ∈ G and x ∈ X where p ( x ) = 0 , if p ( gx ) = 0 and f ( gx ) = gf ( x ) , h has incorrect equivariance with respect to f at x under transformation g .

̸

̸

Definition 3.7 (Pointwise Extrinsic Equivariance) . For g ∈ G and x ∈ X where p ( x ) = 0 , if p ( gx ) = 0 , h has extrinsic equivariance with respect to f at x under transformation g .

Notice that the definitions of pointwise correct, incorrect, and extrinsic equivariance are mutually exclusive, i.e., a pair ( x, g ) can only have one of the three properties. The pointwise definitions are generalizations of the global Definitions 3.1- 3.3. For example, when pointwise correct equivariance holds for all x ∈ X and g ∈ G , Definition 3.1 is satisfied.

̸

Example 3.8 (Example of Pointwise Correct, Incorrect, and Extrinsic Equivariance) . Consider the same binary classification task in Example 3.4. Figure 2 shows f ( x ) , g ( x ) , and four subsets of X where pointwise correct, incorrect, or extrinsic holds. For x in the correct section (green), p ( x ) > 0 , p ( gx ) > 0 , f ( x ) = f ( gx ) . For x in the incorrect sections (red), p ( x ) > 0 , p ( gx ) > 0 , f ( x ) = f ( gx ) . For x in the extrinsic section (blue), p ( x ) > 0 , p ( gx ) = 0 .

Definition 3.9 (Correct, Incorrect, and Extrinsic Sets) . The Correct Set C ⊆ X × G is a subset of X × G where pointwise correct equivariance holds for all ( x, g ) ∈ C . Similarly, the Incorrect Set I and the Extrinsic Set E are subsets where incorrect equivariance or extrinsic equivariance holds for all elements in the subset. Denote U ⊆ X × G as the Undefined Set where ∀ ( x, g ) ∈ U, p ( x ) = 0 . By definition we have X × G = C ⨿ I ⨿ E ⨿ U , where ⨿ denotes a disjoint union.

4 Approximation Error Lower Bound from Incorrect Equivariance

Studying the theoretical error lower bound of an equivariant network is essential for model selection, especially when incorrect equivariance exists. Wang et al. [58] prove an error lower bound for an incorrect equivariant network, but their setting is limited to a classification task in the global situation of Definition 3.2 with a discrete group and an invariant density function. In this section, we find the lower bound of err( h ) for an equivariant model h in a general setting. To calculate such a lower bound, we first define the fundamental domain F of X . Let d be the dimension of a generic orbit of G in X and n the dimension of X . Let ν be the ( n -d ) dimensional Hausdorff measure in X .

Definition 4.1 (Fundamental Domain) . A closed subset F of X is called a fundamental domain of G in X if X is the union of conjugates 2 of F , i.e., X = ⋃ g ∈ G gF, and the intersection of any two conjugates has 0 measure under ν .

̸

We assume further that the set of all x which lie in any pairwise intersection ⋃ g 1 F = g 2 F ( g 1 F ∩ g 2 F ) has measure 0 under ν . Let Gx = { gx : g ∈ G } be the orbit of x , then X can be written as the union of the orbits of all points in the fundamental domain F as such X = ⋃ x ∈ F Gx .

4.1 Lower Bound for Classification

We first show the lower bound of the error err cls ( h ) (Equation 1) given the invariant constraint in h : h ( gx ) = h ( x ) , g ∈ G . In this section, the codomain Y of f is a finite set of possible labels. Since h is G -invariant, h has the same output for all inputs in an orbit Gx . We call the label that causes the minimal error inside the orbit the majority label 3 , and define the error in the orbit as the total dissent .

Definition 4.2 (Total Dissent) . For the orbit Gx of x ∈ X , the total dissent k ( Gx ) is the integrated probability density of the elements in the orbit Gx having a different label than the majority label

̸

We can also lift the integral to G itself by introducing a factor α ( x, g ) to account for the Jacobian of the action map and size of the stabilizer of x . (See Appendix A.)

̸

̸

Theorem 4.3. err( h ) is lower bounded by ∫ F k ( Gx ) dx .

Proof. Rewriting the error function of Equation 1, we have

̸

̸

using iterated integration (Appendix B) and Definition 4.1. We assume the measure of F ∩ gF is 0. Since h ( z ) can only have a single label in orbit Gx , we can lower bound the inside integral as

̸

̸

We obtain the claim by integrating over F . Notice that this is a tight lower bound assuming universal approximation. That is, there exists h which realizes this lower bound.

We can express the total dissent in terms of the Incorrect Set I (Definition 3.9).

Proposition 4.4. k ( Gx ) = min x ′ ∈ ( Gx ) + ∫ G p ( gx ′ ) ✶ (( x ′ , g ) ∈ I ) α ( x ′ , g ) dg , where ( Gx ) + = { x 0 ∈ Gx | p ( x 0 ) > 0 } .

Proof. Consider Equation 3, since the minimum over y is obtained for y = f ( x ′ ) for some x ′ ∈ Gx such that p ( x ′ ) > 0 (i.e., x ′ ∈ ( Gx ) + ),

̸

Since x ′ ∈ Gx , then Gx ′ = Gx and we have k ( Gx ) = k ( Gx ′ ) . Thus,

̸

Example 4.5 (Lower bound example for a binary classification task using Proposition 4.4) .

Let f : X → { 0 , 1 } be a binary classification function on X = { x 0 , x 1 , x 2 , x 3 } . Let G = C 2 = { e, r } be the cyclic group of order two that permutes the elements in X . Figure 3 shows X , the label for each x ∈ X , and how e, r ∈ G acts on x ∈ X . { x 0 , x 3 } forms a fundamental domain F , and there are two orbits: Gx 0 = { x 0 , x 1 } and Gx 2 = { x 2 , x 3 } . Since both X and G are discrete and g ∈ G acts on X through permutation, The lower bound can be written as err( h ) ≥ ∑ x ∈ F min x ′ ∈ ( Gx ) + ∑ g ∈ G p ( gx ′ ) ✶ (( x ′ , g ) ∈ I ) . We can then calculate ∑ g ∈ G p ( gx ′ ) ✶ (( x ′ , g ) ∈ I ) for x ′ ∈ X : x 0 : 0 . 4 , x 1 : 0 . 3 , x 2 : 0 , x 3 : 0 . Taking the min over each orbit we have k ( Gx 0 ) = 0 . 3 , k ( Gx 2 ) = 0 . Taking the sum over F = { x 0 , x 3 } we obtain err( h ) ≥ 0 . 3 .

OrbitO

r

MajorityLabelRatio

&

(a0x)

e

(xo,r)∈I

C1

e

女

(aIx)

X

（x1,r)∈I

XEX

p(xo) = 0.3

r

p(a1) = 0.4

Gx

Orbit1

r

F

女

(x2,r)I

Correct Ratio

(22,e)

e

C2

33

e

(a2)

(3，r)I

p(α2) = 0.1

r

p(x3) = 0.2

y=0

y=1

Symmetry:horizontal-invariant

(x,9) I→(x,9)∈ I

Example 4.6 (Lower bound example for a multi-class classification task using Proposition 4.4) . Consider a multi-class classification task f : R 2 → Y with n = | Y | classes. For x = ( u, v ) ∈ [0 , 1] 2 then p ( u, v ) = 1 and otherwise p ( u, v ) = 0 ; i.e., the support of p is a unit square. Let G denote the group of translations in the u -direction and h a G -invariant network. In a data distribution illustrated in Figure 4, we compute the lower bound for err( h ) . Consider a fundamental domain F (brown line in Figure 4). In the blue area, there is one label across the orbit (i.e., the horizontal

line), meaning ∀ g ∈ G, ( x ′ , g ) ∈ C , yielding Proposition 4.4 equals 0. For points in the yellow area, the majority label is yellow. This means that for g ∈ G such that gx is in yellow, ( x, g ) ∈ C ; for other g ∈ G, ( x, g ) ∈ I . Consequently, Proposition 4.4 is equivalent to the combined green and pink lengths. Taking the integral over F (Theorem 4.3), the lower bound equals the green and pink area ( I in Figure 4). We define correct ratio ( c ) as the blue area's height and majority label ratio ( m ) as the yellow area's length. Adjusting c and m transitions incorrect to correct equivariance, leading to err( h ) ≥ area ( I ) = (1 -c ) × (1 -m ) . Appendix H.2 shows an experiment where the empirical result matches our analysis.

Lower Bound When G is Finite and The Action of G is Density Preserving. In this section, we consider the lower bound in Theorem 4.3 when G is finite and the action of G is density preserving, i.e., p ( gx ) = p ( x ) . Let ( Gx ) y = { z ∈ Gx | f ( z ) = y } be a subset of Gx with label y . Define Q ( x ) = (max y ∈ Y | ( Gx ) y | ) / | Gx | , which is the fraction of data in the orbit Gx that has the majority label. Denote Q = {Q ( x ) : x ∈ X } the set of all possible values for Q . Consider a partition of X = ∐ q ∈ Q X q where X q = { x ∈ X : Q ( x ) = q } . Define c q = P ( x ∈ X q ) = | X q | / | X | .

Proposition 4.7. The error lower bound err( h ) ≥ 1 -∑ q qc q from Wang et al. [58] (Proposition 4.1) is a special case of Theorem 4.3.

Proof in Appendix C. The proposition shows Theorem 4.3 is a strict generalization of [58, Prop 4.1].

4.2 Lower Bound for Invariant Regression

In this section, we give a lower bound of the error function err reg ( h ) (Equation 1) in a regression task given that h is invariant, i.e., h ( gx ) = h ( x ) for all g ∈ G . Assume Y = R n . Denote by p ( Gx ) = ∫ z ∈ Gx p ( z ) dz the probability of the orbit Gx . Denote by q ( z ) = p ( z ) p ( Gx ) the normalized probability density of the orbit Gx such that ∫ Gx q ( z ) dz = 1 . Let E Gx [ f ] be the mean of function f on the orbit Gx defined, and let V Gx [ f ] be the variance of f on the orbit Gx ,

Theorem 4.8. err( h ) ≥ ∫ F p ( Gx ) V Gx [ f ] dx .

Proof. The error function (Equation 1) can be written as:

Denote e ( x ) = ∫ Gx p ( z ) || f ( z ) -h ( z ) || 2 2 dz . Since h is G -invariant, there exists c ∈ R n such that h ( z ) = c for all z ∈ Gx . Then e ( x ) can be written as e ( x ) = ∫ Gx p ( z ) || f ( z ) -c || 2 2 dz. Taking the derivative of e ( x ) with respect to c and setting it to 0 gives c ∗ , the minimum of e ( x ) , c ∗ = ∫ Gx p ( z ) f ( z ) dz ∫ Gx p ( z ) dz = E Gx [ f ] . Substituting c ∗ into e ( x ) we have

We can obtain the claim by taking the integral of e ( x ) over the fundamental domain F .

4.3 Lower Bound for Equivariant Regression

We now prove a lower bound for err( h ) in a regression task given the model h is equivariant, that is, h ( ρ X ( g ) x ) = ρ Y ( g ) h ( x ) where g ∈ G,ρ X and ρ Y are group representations associated with X and Y . We will denote ρ X ( g ) x and ρ Y ( g ) y by gx and gy , leaving the representation implicit. Assume Y = R n and α ( x, g ) is the same as in equation 3. Let Id be the identity. Define a matrix Q Gx ∈ R n × n and q ( gx ) ∈ R n × n so that ∫ G q ( gx ) dg = Id by

O

f(co)=(2,8)

h(c1)=rh(co)

f（gzo）

= (-4,4)

f(c1)=(-4,2)

1=r0

vr

2f（r²x0)

h(co);=(4,4)

f(x2)=(-8,-2)

O

h(x)=(-2,-1)

f(x3)=(4,-2)

h(c2) =r²h(c0) h(c3)=²h(xo)

=(-4,-4)

= (4,-4)

(a)

(b)

(c)

(d)

Here, for simplicity, we assume Q Gx is an invertible matrix. (See Appendix D for general case).

If f is equivariant, g -1 f ( gx ) is a constant for all g ∈ G . Define E G [ f, x ]

Theorem 4.9. The error of h has lower bound err( h ) ≥ ∫ F ∫ G p ( gx ) || f ( gx ) -g E G [ f, x ] || 2 2 α ( x, g ) dgdx .

See Appendix D for the proof. Intuitively, E G [ f, x ] is the minimizer obtained by taking the mean of all inversely transformed f ( x ) for all x in the orbit, see Figure 5cd and Example 4.11 below.

Corollary 4.10. Denote p ( Gx ) = ∫ Gx p ( z ) dz . Denote q x : g ↦→ q ( gx ) . Define G -stabilized f as f x : g ↦→ g -1 f ( gx ) . When ρ Y is an orthogonal representation ρ Y : G → O( n ) ⊂ GL ( n ) , q x is a probability density function on G . Denote the variance of f x as V G [ f x ] where g ∼ q x . The error has a lower bound err( h ) ≥ ∫ F p ( Gx ) V G [ f x ] dx .

See Appendix E for the proof. Notice that Corollary 4.10 is a generalization of Theorem 4.8. That is, Theorem 4.8 can be recovered by taking ρ Y ( g ) = Id (See the proof in Appendix F).

Example 4.11 (Lower bound example of a regression task) . Consider a regression problem where X = { x 0 , x 1 , x 2 , x 3 } and Y = R 2 . Assume p is uniform density. The cyclic group G = C 4 = { e, r, r 2 , r 3 } (where e = 0 rotation and r = π/ 2 rotation) acts on X through x 1 = rx 0 ; x 2 = rx 1 ; x 3 = rx 2 ; x 0 = rx 3 (i.e., there is only one orbit Gx = X ). g ∈ G acts on y ∈ Y through ρ Y ( g ) = ( cos g -sin g sin g cos g ) . Figure 5a shows the output of f ( x ) , ∀ x ∈ X . First , consider a G -invariant network h . Since there is only one orbit, Theorem 4.8 can be simplified as: err( h ) ≥ V X [ f ] , the variance of f over X . This can be calculated by first taking the mean of f ( x ) then calculating the mean square error (MSE) from all x to the mean (Figure 5b). Second , consider a G -equivariant network h . Since G is discrete, gx permutes the order of X , ρ Y is an orthogonal representation, and there is only one orbit, Corollary 4.10 can be written as err( h ) ≥ V G [ f x ] , the variance of G -stabilized f . First, to calculate E G [ f x ] , let x = x 0 , we stabilize g from f by g -1 f ( gx ) for all g ∈ G , then take the mean (Figure 5c). We can then find V G [ f x ] by calculating the MSE between f ( x ) and transformed mean g E G [ f x ] (Figure 5d). Appendix H.3 shows an experiment in this example's environment.

5 Harmful Extrinsic Equivariance

Wang et al. [58] demonstrate that extrinsic equivariance, where the symmetry imposed on the model leads to out-of-distribution data with respect to the input distribution, can lead to a higher performance on the original training data. In this section, we argue that this is not necessarily true in all cases, and there can exist scenarios where extrinsic equivariance can even be harmful to the learning problem.

Consider a binary classification task where the domain is discrete and contains only a set of four points S ⊂ R 3 , and their labels are either {-1 , +1 } as shown in Figure 6a. We consider the probability density p to be uniform for this domain, i.e., p ( x ) = 1 / 4 for the four points S , and p = 0 elsewhere.

This domain is used for both model training and testing so there is no distribution shift. We consider two model classes, F N , the set of all linear models, and F E , the set of all linear models which are invariant with respect to the cyclic group C 2 = { 1 , g } , where g ( x 1 , x 2 , x 3 ) = ( x 1 , x 2 , -x 3 ) . F N corresponds to an unconstrained or a non-equivariant model class and F E corresponds to an extrinsically equivariant class for this domain. For the labeling shown in Figure 6, the hyperplane x 3 = 0 correctly classifies all samples and is contained in F N . However, a function f e ∈ F E is equivalent to a linear classifier on R 2 and effectively sees the data as Figure 6b 4 . This exclusive-or problem does not admit a linear solution (it can be correct for at most 3 points).

Concretely, we can compute the empirical Rademacher complexity, a standard measure of model class expressivity, for nonequivariant and extrinsically equivariant model classes and show that F E has lower complexity than F N . Recall that empirical Rademacher complexity is defined as R S ( F ) = E σ [ sup f ∈F 1 m ∑ m i =1 σ i f ( x i ) ] , where S is the set of m samples and σ = ( σ 1 , . . . , σ m ) ⊤ , σ i ∈ {-1 , +1 } are independent uniform Rademacher random variables, and x i is the i -th sample. As there exists some linear function f n ∈ F N that fully classifies S for any combination of labels, R S ( F N ) = 1 . For the extrinsic equivariance case, of the 16 possible label combinations, there are two cases where f e ∈ F E can at most classify 3 out of 4 points correctly, and thus R S ( F E ) = 31 32 < R S ( F N ) (see Appendix G

(a) Data

(b) Transformed

data

(c)

C

2

-equivariant

view

for the calculations). This illustrates that in certain cases, extrinsic equivariance can lead to lower model expressivity than no equivariance and thus be harmful to learning.

6 Experiments

We perform experiments to validate our theoretical analysis on both the lower bounds (Section 4) and the harmful extrinsic equivariance (Section 5). We find that our bounds accurately predict empirical model error. In addition to the experiments in this section, Appendix H.2 shows an experiment verifying our classification bound (Theorem 4.3) and Appendix H.3 shows an experiment verifying our regression bound (Theorem 4.8 and 4.9). The experiment details are in Appendix I.

6.1 Swiss Roll Experiment

We first perform an experiment in a vertically separated Swiss Roll data distribution, see Figure 7a 5 . This example, similar to that in Section 5, demonstrates that a C 2 -invariant model effectively 'flattens' the z -dimension of the data so it must learn the decision boundary between two spirals (Figure 7b), whereas the non-equivariant model only needs to learn a horizontal plane to separate the classes, a significantly easier task. Besides the extrinsic data distribution, we consider two other data distributions shown in Figure 7c and Figure 7d, where a C 2 -invariant model will observe incorrect and correct equivariance due to the mismatched and matched data labels in the two z planes.

We combine data from all three distributions in various proportions to test the performance of a z -invariant network (INV) with a baseline unconstrained network (MLP). Let c be the correct ratio, the proportion of data from the correct distribution. Define the incorrect ratio i and extrinsic ratio e similarly. We consider all c, i, e that are multiples of 0.125 such that c + i + e = 1 . Figure 7ef shows some example data distributions. Relative to INV, this mixed data distribution has partial correct, incorrect, and extrinsic equivariance, which is not fully captured in prior work [58]. Based on Proposition 4.4, we have k ( Gx ) = 0 . 5 for x drawn from the incorrect distribution, and k ( Gx ) = 0 otherwise. Since the data is evenly distributed, we can calculate the error lower bound err( h ) ≥ 0 . 5 i .

(a) Swiss Roll

Data

(b)

C

2

-Invariant

View

(c) Incorrect

(d) Correct

(e)

c

=

.

5

, i

=

.

5

, e

= 0

(f)

c

=

.

5

, i

=

0

, e

=

.

5

0.4

0.2

0.0

0.50

0.2

0.50

0.50

0.50

0.50

0.25

0.25

0.25

0.25

0.25

05'0-

0.25

0.00

K

0.25

9.500.50

0.25

0.00

-0.4

0.4

0.2

0.0

0.2

0.4

0.50

0.00

0.25

0.500.50

0.25

0.00

T

0.50

52'0

0.00

X

0.25

0.50~0.50

0.25

0.00

05'0-

0.00

X

0.25

0.5o~0.50

0.25

0.00

05'0-

0.00

0.25

0.50~0.50

0.25

0.00

Results. Figure 8a shows the test success rate of INV compared with MLP when e and c vary with i = 0 . When e increases, the performance of INV decreases while the performance of MLP shows an inverse trend, demonstrating that extrinsic equivariance is harmful in this experiment. Figure 8b shows the performance of INV and MLP when c and i vary while e = 0 . The green line shows the upper bound of the test success rate ( 1 -0 . 5 i ). The experimental result matches our theoretical analysis quite closely. Notice that when c increases, there is a bigger gap between the performance of the network and its theoretical upper bound, since classification in the correct distribution is a harder task. Appendix H.1 shows the complete results of this experiment.

6.2 Digit Classification Experiment

In this experiment, we apply our theoretical analysis to a realistic digit classification task using both the printed digit dataset [16] and the MNIST handwritten digit dataset [15]. We compare a D 4 -invariant network ( D 4 ) with an unconstrained CNN. In the printed digit classification, D 4 exhibits incorrect equivariance for 6 and 9 under a π rotation. Using Theorem 4.3, we can calculate a lower bound of error for D 4 at 10%. However, as shown in Table 1 (top), the experimental results indicate that the actual performance is slightly better than predicted by the theory. We hypothesize that this discrepancy arises because a rotated 9 differs slightly from a 6 in some fonts. We conduct a similar experiment using the MNIST handwritten digit dataset (Table 1 bottom), where D 4 achieves even better performance in classifying 6 and 9. This improvement is likely due to the more distinguishable handwriting of these digits, although the performance still underperforms the CNN as incorrect equivariance persists. It is important to note that there is a significant decrease in performance for D 4 when classifying 2/5 and 4/7 compared to the CNN. This is because a vertical flip results in incorrect equivariance when classifying handwritten 2/5, and a similar issue arises for 4/7 under a π/ 2 rotation followed by a vertical flip (notice that Weiler and Cesa [60] make a similar observation). These experiments demonstrate that our theory is useful not only for calculating the performance bounds of an equivariant network beforehand, but also for explaining the suboptimal performance of an equivariant network, thereby potentially assisting in model selection.

6.3 Robotic Experiment

In this experiment, we evaluate our theory in behavior cloning in robotic manipulation. We first preform an experiment where the problem is a mixture of correct and incorrect equivariance for a D 1 -equivariant policy network ( D 1 ) where the robot's action will flip when the state is flipped test sr

(a)

(b)

1.0

1.0

0.8 -

0.8 -

0.6

test sr

0.6 -

0.4 -

0.4

0.2

0.2

INV UB

INV

INV

0.0

MLP

0.0-

MLP

e=1, c=0

e=0.5, c=0.5

e=0, c=1

c=1, i=0

c=0.5, i=0.5

c=0, i=1

extrinsic -> correct

correct -> incorrect

StackingwithCorrectEqui.

SortingwithIncorrectEqui.：SortingwithExtrinsicEqui.

ProbabilityofStacking

ProbabilityofSorting

ColorofLeft

ColorofRight

(correctequi.):c

(incorrect equi.):1-C

1.0 -

- 80

test success rate

0.6-

0.4

0.2 -

D1

CNN

0.0 -

D1 UB

0.2

0.4

0.6

0.8

correct ratio

horizontally. Specifically, the environment contains two possible tasks (Figure 9 left). Stacking requires the robot to stack a green triangle on top of a blue cube. Here flip equivariance is correct. Sorting requires the robot to push the red cube to the left and the yellow triangle to the right. Here flip equivariance is incorrect because the robot should not sort the objects in an opposite way when the state is flipped (in other words, D 1 cannot distinguish left and right). We vary the probability c of the stacking task (correct ratio) in the task distribution, and compare the performance of D 1 versus a standard CNN policy. If we view the sorting task as a binary classification task, we can calculate an upper bound of performance for D 1 using Theorem 4.3: 0 . 5 + 0 . 5 c . Figure 10 shows the result. Notice that the performance of D 1 closely matches the theoretical upper bound, while the performance of CNN remains relatively stable for all Stacking-Sorting distributions.

We further evaluate D 1 in a sorting task with harmful extrinsic equivariance. Here, the goal for the robot is the same as sorting above (push the red cube left and the yellow triangle right), however, the left and right sides of the workspace can now be differentiated by gray-scale colors. The shades of gray are discretized evenly into n bins, where the left side's color is randomly sampled from the odd-numbered bins, and the right side's color is randomly sampled from the even-numbered bins (Figure 9 right). The different color distributions of the left and right sides make D 1 extrinsically equivariant, but it needs to learn the color distribution to distinguish left and right (while CNN can distinguish left and right directly). We set n = 10 , and D 1 achieves 71 . 5 ± 1 . 6% test success rate, while CNN achieves 99 . 5 ± 0 . 5% , demonstrating that the D 1 extrinsic equivariance is harmful in this task. See Appendix I.5 for the details of the robot experiment.

7 Discussion

This paper presents a general theory for when the symmetry of the ground truth function and equivariant network are mismatched. We define pointwise correct, incorrect, and extrinsic equivariance, generalizing prior work [58] to include continuous mixtures of the three extremes. We prove error lower bounds for equivariant networks applied to asymmetric tasks including classification, invariant regression, and equivariant regression without the assumption of invariant data density. Our work discusses the potential disadvantage of extrinsic equivariance, and provides experiments that validate our theoretical analysis. The major limitation of this paper is that our theoretical lower bounds require domain knowledge like the density function over the domain. In future work, we will develop easy-toapply model selection tools using our theory. Another future direction is theoretically understanding when extrinsic equivariance is helpful or harmful and analyzing the effect of extrinsic equivariance on the decision boundary of an equivariant network.

Acknowledgments

This work is supported in part by NSF 1724257, NSF 1724191, NSF 1763878, NSF 1750649, NSF 2107256, NSF 2134178, NSF 2312171, and NASA 80NSSC19K1474.

References

Yaser S Abu-Mostafa. Hints and the vc dimension. Neural Computation , 5(2):278-288, 1993.

Brandon Anderson, Truong Son Hy, and Risi Kondor. Cormorant: Covariant molecular neural networks. Advances in neural information processing systems , 32, 2019.

Fabio Anselmi, Georgios Evangelopoulos, Lorenzo Rosasco, and Tomaso Poggio. Symmetryadapted representation learning. Pattern Recognition , 86:201-208, 2019.

Kenneth Atz, Francesca Grisoni, and Gisbert Schneider. Geometric deep learning on molecular representations. Nature Machine Intelligence , 3(12):1023-1032, 2021.

Arash Behboodi, Gabriele Cesa, and Taco Cohen. A pac-bayesian generalization bound for equivariant networks. arXiv preprint arXiv:2210.13150 , 2022.

Alexander Bogatskiy, Brandon Anderson, Jan Offermann, Marwah Roussi, David Miller, and Risi Kondor. Lorentz group equivariant neural network for particle physics. In International Conference on Machine Learning , pages 992-1002. PMLR, 2020.

Evangelos Chatzipantazis, Stefanos Pertigkiozoglou, Edgar Dobriban, and Kostas Daniilidis. SE(3) -equivariant attention networks for shape reconstruction in function space. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview. net/forum?id=RDy3IbvjMqT .

Shuxiao Chen, Edgar Dobriban, and Jane H Lee. A group-theoretic framework for data augmentation. The Journal of Machine Learning Research , 21(1):9885-9955, 2020.

Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning , pages 2990-2999. PMLR, 2016.

Taco S. Cohen and Max Welling. Steerable CNNs. In International Conference on Learning Representations , 2017. URL https://openreview.net/forum?id=rJQKYt5ll .

Taco S Cohen, Mario Geiger, and Maurice Weiler. A general theory of equivariant cnns on homogeneous spaces. Advances in neural information processing systems , 32, 2019.

Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. http://pybullet.org , 2016-2021.

Rumen Dangovski, Li Jing, Charlotte Loh, Seungwook Han, Akash Srivastava, Brian Cheung, Pulkit Agrawal, and Marin Soljacic. Equivariant self-supervised learning: Encouraging equivariance in representations. In International Conference on Learning Representations , 2021.

Nima Dehmamy, Robin Walters, Yanchen Liu, Dashun Wang, and Rose Yu. Automatic symmetry discovery with lie algebra convolutional network. Advances in Neural Information Processing Systems , 34:2503-2515, 2021.

Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine , 29(6):141-142, 2012.

Kshitij Dhama. Printed numerical digits image dataset. https://github.com/kaydee0502/ printed-digits-dataset , 2021. GitHub Repository.

Bryn Elesedy and Sheheryar Zaidi. Provably strict generalisation benefit for equivariant models. In International Conference on Machine Learning , pages 2959-2969. PMLR, 2021.

Marc Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data. In International Conference on Machine Learning , pages 3165-3176. PMLR, 2020.

Søren Hauberg, Oren Freifeld, Anders Boesen Lindbo Larsen, John Fisher, and Lars Hansen. Dreaming more data: Class-dependent distributions over diffeomorphisms for learned data augmentation. In Artificial intelligence and statistics , pages 342-350. PMLR, 2016.

Geoffrey E Hinton, Alex Krizhevsky, and Sida D Wang. Transforming auto-encoders. In International conference on artificial neural networks , pages 44-51. Springer, 2011.

Haojie Huang, Dian Wang, Robin Walters, and Robert Platt. Equivariant transporter network. In Robotics: Science and Systems , 2022.

Haojie Huang, Dian Wang, Xupeng Zhu, Robin Walters, and Robert Platt. Edge grasp network: A graph-based SE(3) -invariant approach to grasp detection. In International Conference on Robotics and Automation (ICRA) , 2023.

Mingxi Jia, Dian Wang, Guanang Su, David Klee, Xupeng Zhu, Robin Walters, and Robert Platt. Seil: Simulation-augmented equivariant imitation learning. In International Conference on Robotics and Automation (ICRA) , 2023.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

David Klee, Ondrej Biza, Robert Platt, and Robin Walters. I2i: Image to icosahedral projection for SO(3) object reasoning from single-view images. arXiv preprint arXiv:2207.08925 , 2022.

David Klee, Ondrej Biza, Robert Platt, and Robin Walters. Image to sphere: Learning equivariant features for efficient pose prediction. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=_2bDpAtr7PI .

Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In International Conference on Machine Learning , pages 2747-2755. PMLR, 2018.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems , 25, 2012.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equivariance and equivalence. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 991-999, 2015.

Xiaolong Li, Yijia Weng, Li Yi, Leonidas J Guibas, A Abbott, Shuran Song, and He Wang. Leveraging se (3) equivariance for self-supervised category-level object pose estimation from point clouds. Advances in Neural Information Processing Systems , 34:15370-15381, 2021.

Xueyi Liu, Ji Zhang, Ruizhen Hu, Haibin Huang, He Wang, and Li Yi. Self-supervised categorylevel articulated object pose estimation with part-level SE(3) equivariance. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview. net/forum?id=20GtJ6hIaPA .

Clare Lyle, Mark van der Wilk, Marta Kwiatkowska, Yarin Gal, and Benjamin Bloem-Reddy. On the benefits of invariance in neural networks. arXiv preprint arXiv:2005.00178 , 2020.

Kaitlin Maile, Dennis George Wilson, and Patrick Forré. Equivariance-aware architectural optimization of neural networks. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=a6rCdfABJXg .

Giovanni Luca Marchetti, Gustaf Tegnér, Anastasiia Varava, and Danica Kragic. Equivariant representation learning via class-pose decomposition. arXiv preprint arXiv:2207.03116 , 2022.

Haggai Maron, Ethan Fetaya, Nimrod Segol, and Yaron Lipman. On the universality of invariant networks. In International conference on machine learning , pages 4363-4371. PMLR, 2019.

Haggai Maron, Or Litany, Gal Chechik, and Ethan Fetaya. On learning sets of symmetric elements. In International conference on machine learning , pages 6734-6744. PMLR, 2020.

Ning Miao, Tom Rainforth, Emile Mathieu, Yann Dubois, Yee Whye Teh, Adam Foster, and Hyunjik Kim. Learning instance-specific augmentations by capturing local invariances. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 24720-24736. PMLR, 23-29 Jul 2023. URL https://proceedings.mlr.press/v202/miao23a.html .

Arnab Kumar Mondal, Vineet Jain, Kaleem Siddiqi, and Siamak Ravanbakhsh. EqR: Equivariant representations for data-efficient reinforcement learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 15908-15926. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/mondal22a.html .

Artem Moskalev, Anna Sepliarskaia, Ivan Sosnovik, and Arnold W.M. Smeulders. LieGG: Studying learned lie group generators. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=9sKZ60VtRmi .

Chuer Pan, Brian Okorn, Harry Zhang, Ben Eisner, and David Held. Tax-pose: Task-specific cross-pose estimation for robot manipulation. In Conference on Robot Learning , pages 17831792. PMLR, 2023.

Jung Yeon Park, Ondrej Biza, Linfeng Zhao, Jan Willem van de Meent, and Robin Walters. Learning symmetric representations for equivariant world model. In International Conference on Machine Learning , 2022. URL https://arxiv.org/abs/2204.11371 .

Robin Quessard, Thomas Barrett, and William Clements. Learning disentangled representations and group structure of dynamical environments. Advances in Neural Information Processing Systems , 33:19727-19737, 2020.

Cédric Rommel, Thomas Moreau, Joseph Paillard, and Alexandre Gramfort. Cadda: Class-wise automatic differentiable data augmentation for eeg signals. In International Conference on Learning Representations , 2021.

Hyunwoo Ryu, Hong in Lee, Jeong-Hoon Lee, and Jongeun Choi. Equivariant descriptor fields: SE(3)-equivariant energy-based models for end-to-end visual robotic manipulation learning. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=dnjZSPGmY5O .

Akiyoshi Sannai, Masaaki Imaizumi, and Makoto Kawano. Improved generalization bounds of group invariant/equivariant deep networks via quotient feature spaces. In Uncertainty in Artificial Intelligence , pages 771-780. PMLR, 2021.

Uwe Schmidt and Stefan Roth. Learning rotation-aware features: From invariant priors to equivariant descriptors. In 2012 IEEE Conference on Computer Vision and Pattern Recognition , pages 2050-2057. IEEE, 2012.

Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua B Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann. Neural descriptor fields: Se (3)-equivariant object representations for manipulation. In 2022 International Conference on Robotics and Automation (ICRA) , pages 6394-6400. IEEE, 2022.

Anthony Simeonov, Yilun Du, Yen-Chen Lin, Alberto Rodriguez Garcia, Leslie Pack Kaelbling, Tomás Lozano-Pérez, and Pulkit Agrawal. Se (3)-equivariant relational rearrangement with neural descriptor fields. In Conference on Robot Learning , pages 835-846. PMLR, 2023.

Kihyuk Sohn and Honglak Lee. Learning invariant representations with local transformations. In John Langford and Joelle Pineau, editors, Proceedings of the 29th International Conference on Machine Learning (ICML-12) , ICML '12, pages 1311-1318, New York, NY, USA, July 2012. Omnipress. ISBN 978-1-4503-1285-1.

Jure Sokolic, Raja Giryes, Guillermo Sapiro, and Miguel Rodrigues. Generalization error of invariant classifiers. In Artificial Intelligence and Statistics , pages 1094-1103. PMLR, 2017.

Elise van der Pol, Daniel Worrall, Herke van Hoof, Frans Oliehoek, and Max Welling. Mdp homomorphic networks: Group symmetries in reinforcement learning. Advances in Neural Information Processing Systems , 33, 2020.

Robin Walters, Jinxi Li, and Rose Yu. Trajectory prediction using equivariant continuous convolution. arXiv preprint arXiv:2010.11344 , 2020.

Dian Wang, Robin Walters, Xupeng Zhu, and Robert Platt. Equivariant Q learning in spatial action spaces. In 5th Annual Conference on Robot Learning , 2021. URL https: //openreview.net/forum?id=IScz42A3iCI .

Dian Wang, Mingxi Jia, Xupeng Zhu, Robin Walters, and Robert Platt. On-robot learning with equivariant models. In 6th Annual Conference on Robot Learning , 2022. URL https: //openreview.net/forum?id=K8W6ObPZQyh .

Dian Wang, Robin Walters, and Robert Platt. SO(2) -equivariant reinforcement learning. In International Conference on Learning Representations , 2022. URL https://openreview. net/forum?id=7F9cOhdvfk_ .

Dian Wang, Colin Kohler, Xupeng Zhu, Mingxi Jia, and Robert Platt. Bulletarm: An opensource robotic manipulation benchmark and learning framework. In Robotics Research , pages 335-350. Springer, 2023.

Dian Wang, Jung Yeon Park, Neel Sortur, Lawson L.S. Wong, Robin Walters, and Robert Platt. The surprising effectiveness of equivariant models in domains with latent symmetry. In International Conference on Learning Representations , 2023. URL https://openreview. net/forum?id=P4MUGRM4Acu .

Rui Wang, Robin Walters, and Rose Yu. Incorporating symmetry into deep dynamics models for improved generalization. arXiv preprint arXiv:2002.03061 , 2020.

Maurice Weiler and Gabriele Cesa. General e (2)-equivariant steerable cnns. Advances in Neural Information Processing Systems , 32, 2019.

Robin Winter, Marco Bertolini, Tuan Le, Frank Noe, and Djork-Arné Clevert. Unsupervised learning of group invariant and equivariant representations. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=47lpv23LDPr .

Dmitry Yarotsky. Universal approximations of invariant maps by neural networks. Constructive Approximation , 55(1):407-474, 2022.

Linfeng Zhao, Xupeng Zhu, Lingzhi Kong, Robin Walters, and Lawson L.S. Wong. Integrating symmetry into differentiable planning with steerable convolutions. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum? id=n7CPzMPKQl .

Allan Zhou, Tom Knowles, and Chelsea Finn. Meta-learning symmetries by reparameterization. arXiv preprint arXiv:2007.02933 , 2020.

Xupeng Zhu, Dian Wang, Ondrej Biza, Guanang Su, Robin Walters, and Robert Platt. Sample efficient grasp learning using equivariant models. In Robotics: Science and Systems , 2022.

A Integrals on the Group

Fundamental Domains In this paper, we are interested in cases in which the group G is not necessarily discrete but may have positive dimension. We do not assume the fundamental domain has non-empty interior, and thus domain is a misnomer. In this case the conjugates of the fundamental domain gF have measure 0 and the condition that their intersection have measure 0 is vacuous. Instead we assume a stronger condition, that the union of all pairwise intersections ⋃ g 1 = g 2 ( g 1 F ∩ g 2 F ) has measure 0. We also require that F and the orbits Gx are differentiable manifolds such that integrals over X may be evaluated ∫ X f ( x ) dx = ∫ F ∫ Gy f ( z ) dzdy similar to Equation 8 from [18].

Reparameterization Consider the integral

Denote the identification of the orbit Gx and coset space G/G x with respect to the stabilizer G x = { g : gx = x } by a x : G/G x → Gx. Then the integral can be written

We can also lift the integral to G itself

Define α ( g, x ) = ( ∫ G x dh ) -1 ∣ ∣ ∣ ∂a x (¯ g ) ∂ ¯ g ∣ ∣ ∣ . Then

B Iterated Integral

Let X be an n -dimensional space, Definition 4.2 (Equation 2) defines k ( Gx ) as an integral over Gx ⊆ X , which is a m -dimensional sub-manifold of X . In Theorem 4.3, Equation 4 rewrites the error function (Equation 1) as an iterated integral over the orbit Gx and then the fundamental domain F using Definition 4.1. In the discrete group case, m would be 0, Equation 2 is an integral of a 0-form in a 0-manifold, which is a sum:

̸

̸

C Proof of Proposition 4.7

Proof. Consider the integral of probability density inside Gx , for a given y , it can be separated into two groups:

̸

̸

We can then rewrite k ( Gx ) in Equation 2 as:

Letting ( Gx ) y = { x ′ ∈ Gx | f ( x ′ ) = y } = f -1 ( y ) ∩ Gx , Equation 9 can be written as:

Theorem 4.3 can be rewritten as:

The first term in Equation 10 uses the fact that X = ⋃ x ∈ F Gx so the integral of the probability of the orbits of all points in the fundamental domain is the integral of the probability of the input domain X which is 1. The second term of Equation 10 uses p ( gx ) = p ( x ) so the integration of p ( z ) on ( Gx ) y becomes p ( x ) times the range of the limit which is the size of ( Gx ) y , | ( Gx ) y | .

Now consider a partition of F = ∐ q F q where F q = { x ∈ F : (max y ∈ Y | ( Gx ) y | ) / | Gx | = q } . We can rewrite Equation 10 as:

Equation 11 uses the definition of q . Equation 12 separates the integral over F into the partition of F . Equation 13 moves q out from the integral because it is a constant inside the integral. Consider the definition of c q , we have:

Equation 14 uses X q = ⋃ x ∈ F q Gx . Equation 15 uses p ( x ) = p ( gx ) . Now we can write Equation 13 as:

D Proof of Theorem 4.9

Define q ( gx ) ∈ R n × n such that

In particular, q ( gx ) exists when Q Gx is full rank. It follows that Q Gx ∫ G q ( gx ) dg = Q Gx . Moreover, Q Gx and q ( gx ) are symmetric matrix.

Proof. The error function (Equation 1) can be written

Denote e ( x ) = ∫ G p ( gx ) || f ( gx ) -h ( gx ) || 2 2 α ( x, g ) dg . Since h is G -equivariant, for each x ∈ F the value c = h ( x ) ∈ R n of h at x determines the value of h across the whole orbit h ( gx ) = gh ( x ) = gc for g ∈ G . Then e ( x ) can be written

Taking the derivative of e ( x ) with respect to c we have

Setting ∂e ( x ) /∂c = 0 we can find an equation for c ∗ which minimizes e ( x )

Substituting c ∗ into Equation 17 we have

The term ∫ G c ∗ T Q Gx q ( gx ) g -1 f ( gx ) dg could be simplified as

Notice that Q Gx , and q ( gx ) are symmetric matrix

Thus Equation 19 becomes

Taking the integral over the fundamental domain F we have

E Proof of Corollary 4.10

Proof. When ρ Y is an orthogonal representation, we have ρ Y ( g ) T ρ Y ( g ) = I n , i.e., the identity matrix. Then q ( gx ) can be written as q ( gx ) = s ( gx )Id where s ( gx ) is a scalar. Since ∫ G q ( gx ) dg = Id , we can re-define q ( gx ) to drop Id and only keep the scalar, then q x ( g ) can be viewed as a probability density function of g because now ∫ G q x ( g ) = 1 .

With q x ( g ) being the probability density function, E G [ f, x ] (Equation 6) naturally becomes the mean E G [ f x ] where g ∼ q x .

∫ 2

Since ρ Y ( g ) ρ Y ( g ) = I n , we have

From Equation 5 we have p ( gx ) α ( x, g ) = Q Gx q ( gx ) . Substituting in Equation 22 we have

Since Q Gx = ∫ G p ( gx ) α ( a, g ) dg when ρ Y ( g ) T ρ Y ( g ) = I n , we have

Now consider Q Gx (Equation 5), when ρ Y ( g ) T ρ Y ( g ) = I n , it can be written

Replacing Q Gx with p ( Gx ) in Equation 23 then taking the integral of e ( x ) over the fundamental domain gives the result.

F Lower Bound of Equivariant Regression when ρ Y = Id

Proposition F.1. When ρ Y = Id , the error of h has lower bound err( h ) ≥ ∫ F p ( Gx ) V Gx [ f ] dx , which is the same as Theorem 4.8.

Proof. Consider Equation 5, when ρ Y ( g ) = Id , we have

Exchange the integration variable using z = gx we have

Consider E G [ f x ] = ∫ G q x ( g ) g -1 f ( gx ) dg . When ρ Y ( g ) = Id , it becomes

Substituting q ( gx ) with Equation 5, considering ρ Y ( g ) = Id , we have

Exchange the integration variable using z = gx we have

Substituting Equation 24 we have

Similarly, we can proof V G [ f x ] = V Gx [ f ] , thus when ρ Y = Id , Corollary 4.10 is Theorem 4.8.

Non-equivariant model class

C

2

-equivariant model class

σ

⊤

sup

f

n

∈F

N

1

m

∑

m

i

=1

σ

i

f

n

(

x

i

)

sup

f

n

∈F

N

1

m

∑

m

i

=1

σ

i

f

n

(

x

i

)

[

-

1

,

-

1

,

-

1

,

-

1]

1

1

[

-

1

,

-

1

,

-

1

,

+1]

1

1

[

-

1

,

-

1

,

+1

,

-

1]

1

1

[

-

1

,

-

1

,

+1

,

+1]

1

0

.

75

[

-

1

,

+1

,

-

1

,

-

1]

1

1

[

-

1

,

+1

,

-

1

,

+1]

1

1

[

-

1

,

+1

,

+1

,

-

1]

1

1

[

-

1

,

+1

,

+1

,

+1]

1

1

[+1

,

-

1

,

-

1

,

-

1]

1

1

[+1

,

-

1

,

-

1

,

+1]

1

1

[+1

,

-

1

,

+1

,

-

1]

1

1

[+1

,

-

1

,

+1

,

+1]

1

1

[+1

,

+1

,

-

1

,

-

1]

1

0

.

75

[+1

,

+1

,

-

1

,

+1]

1

1

[+1

,

+1

,

-

1

,

+1]

1

1

[+1

,

+1

,

+1

,

+1]

1

1

R

S

1

31

32

(a) Correct

(b) Incorrect

(c) Extrinsic

(d)

Extrinsic

Invariant

2.0

1.5

1.0 -

Z

0.5 -

F 0°0

0

0

0

0.5

1.5

2.0

1.5

2.0

1.5

2.0

1.0

000-

0.0

0.5

1.0

00001Z-

0.0

0.5

1.0

0000010

0°0

0.5

1.0

1.5

X

0.0

0.5

1.0

1.5

2.0 2.0

1.5

1.0

-0.5y

X

0.51.0

1.5

-1.0

0.5

y

X

0.5

1.0

1.5

2.0 2.0

1.5

1.0

-0.5y

2.0 -

-2.0-1.5-1.00.50.00.5

1.0

1.5

2.0

View

G Rademacher Complexity of Harmful Extrinsic Equivariance Example

Let S = { x 1 , x 2 , x 3 , x 4 } , where the labels are y 1 , y 2 = +1 and y 3 , y 4 = -1 . We consider two model classes F N , the set of all linear models, and F E , the set of all linear models equivariant to C 2 , and compute their empirical Rademacher complexity on S .

For the data S , an extrinsically equivariant linear model class has lower empirical Rademacher complexity than its unconstrained linear counterpart, demonstrating that extrinsic equivariance can be harmful to learning.

H Additional Experiments

H.1 Swiss Roll Experiment

Figure 11 and Figure 12 show the actual data distribution for the Swiss Roll experiment in Section 6.1. In the incorrect distribution, the data in the two z planes form two spirals with different labels but the same shape. The equivariance is incorrect because if we translate one spiral to the other spiral's plane, they will overlap but their labels are different. In the correct distribution, there are two different 'dashed' spirals copied into two z -planes. The equivariance is correct because after a z -translation, both the data and their labels exactly overlap. In all three cases, we assume the data has a uniform distribution. Figure 13b shows the ternary plot of MLP for all different c, ir, er , where the performance of MLP decreases as the correct ratio increases. Figure 13a shows an inverse trend: the

(a)

c

=

.

5

, i

=

.

5

, e

= 0

(b)

c

=

.

5

, i

= 0

, e

=

.

5

(c)

c

= 0

, i

=

.

5

, e

=

.

5

(d)

c

=

.

5

, i

=

.

25

, e

=

.

25

Z

0

Q

1.5

2.0

1.5

2.0

1.5

2.0

1.5

2.0

-2.0_1.5_1.0_0.5

0.0

0.5

1.0

2.0_1.5_1.0_0.5 c

0.0

0.5

1.0

-2.0_1.5_1.0_0.5

0°0

0.5

1.0

-2.0_1.5_1.0_0.5

0°0

0.5

1.0

X

0.0

0.5

1.0

1.5

2.0 2.0

1.5

1.0

-0.5y

X

0.0

0.5

1.0

1.5

2.0 2.0

1.5

1.0

0.5

X

0.0

0.5

1.0

1.5

2.0 2.0

1.5

1.0

0.5y

X

0.0

0.5

1.0

1.5

2.0 2.0

1.5

-1.0

-0.5y

(a)

(b)

INV

MLP

incorrect

incorrect

0.00

1.00

1.0

0.00

1.00

1.0

0.25

0.75

0.9

0.25

0.75

0.9

0.7

0.7

0.75

0.25

0.75

0.25

0.6

0.6

extrinsic

0.00

1.00

0.25

0.50

7

0.75

1.00

0.00

correct

extrinsic

1.00

Q.00

0.25

0.50

0.75

1.00

0.00

correct

correct ratio

correct ratio

performance of INV increases as the correct ratio increases. Moreover, both extrinsic and incorrect equivariance harms the performance of INV, but incorrect equivariance is more devastating because the error is limited by a theoretical lower bound.

H.2 Square Experiment

We consider the environment shown in Example 4.6. We vary m ∈ { 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 , 1 } and c ∈ { 0 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 , 1 } . We train an u -invariant network and evaluate its test performance with the theoretical lower bound err( h ) ≥ (1 -c ) × (1 -m ) . Figure 14 shows the test error of the trained network compared with the theoretical lower bound. The highest difference is below 3%, demonstrating the correctness of our theory.

H.3 Regression Experiment

In this experiment, we validate our theoretical error lower bound for invariant and equivariant regression (Theorem 4.8, 4.9) in an environment similar to Example 4.11. Consider a regression task f : R ×X → R 2 given by ( θ, x ) ↦→ y , where X = { x 0 , x 1 , x 2 , x 3 } . The group g ∈ G = C 4 = { e, r, r 2 , r 3 } acts on ( θ, x ) by g ( θ, x ) = ( θ, gx ) through permutation: x 1 = rx 0 ; x 2 = rx 1 ; x 3 =

rx 2 ; x 0 = rx 3 . Let r k ∈ G acts on y by ρ Y ( g ) = ( cos g -sin g sin g cos g ) where g = kπ/ 2 . Note that fixing a single value of θ gives Example 4.11; in other words, this experiment has infinitely many orbits where each orbit is similar to Example 4.11.

̸

We generate random polynomial function f that is not equivariant, i.e., ∃ ( θ, x ) s.t. g · f ( θ, x ) = ρ Y ( g ) y . Then we try to fit f using a G -invariant network and a G -equivariant network. We measure their error compared with the theoretical lower bound given by Theorem 4.8 and 4.9. As is shown in Table 2, both the invariant network and the equivariant network achieve an error rate nearly the same as our theoretical bound. The empirical error is slightly higher than the theoretical error due to the neural network fitting error. Please refer to I.4 for more experiment details.

I Experiment Details

This section describes the details of our experiments. All of the experiment is performed using a single Nvidia RTX 2080 Ti graphic card.

I.1 Swiss Roll Experiment

In the Swiss Roll Experiment in Section 6.1, we use a three-layer MLP for the unconstrained network. For the z -invariant network, we use a network with two DSS [37] layers to implement the z -invariance, each containing two FC layers. We train the networks using the Adam [24] optimizer with a learning rate of 10 -3 . The batch size is 128. In each run, there are 200 training data, 200 validation data, and 200 test data randomly sampled from the data distribution. The network is trained for a minimal of 1000 epochs and a maximum of 10000 epochs, where the training is terminated after there is no improvement in the classification success rate in the validation set for a consecutive of 1000 epochs. We report the test success rate of the epoch model with the highest validation success rate.

I.2 Square Experiment

In the Square Experiment in Section H.2, we use a network with two DSS [37] layers to implement the horizontal invariance, where each layer contains two FC layers. We train the networks using the Adam [24] optimizer with a learning rate of 10 -3 . The batch size is 128. In each run, there are 1000 training data, 200 validation data, and 200 test data randomly sampled from the data distribution. The network is trained for a minimal of 1000 epochs and a maximum of 10000 epochs, where the training is terminated after there is no improvement in the classification success rate in the validation set for a consecutive of 1000 epochs. We report the test success rate of the epoch model with the highest validation success rate.

I.3 Digit Classification Experiment

In the Digit Classification Experiment in Section 6.2, we use two similar five-layer convolutional networks for the D 4 -invariant network and the CNN, where the D 4 -invariant network is implemented using the e2cnn package [60]. Both networks have the similar amount of trainable parameters. We train the networks using the Adam [24] optimizer with a learning rate of 5 × 10 -5 and weight decay

Workspace

Workspace

Observation

Observation

Flip

D-Equivariant Policy

(x, y)

(x, -y)

θ

0

Flip

Z

Z

of 10 -5 . The batch size is 256. In each run, there are 5000 training data, 1000 validation data, and 1000 test data randomly sampled from the data distribution. The network is trained for a minimal of 50 epochs and a maximum of 1000 epochs, where the training is terminated after there is no improvement in the classification success rate in the validation set for a consecutive of 50 epochs. We report the test success rate of the epoch model with the highest validation success rate.

I.4 Regression Experiment

In the regression experiment, we validate our theoretical error lower bound for invariant and equivariant regression (Theorem 4.8, 4.9) by comparing empirical network fitting error and the theoretical fitting error of a function f . Specifically, the function f maps a distance θ and an index x pair to a vector y :

where X = { x 0 , x 1 , x 2 , x 3 } . The group g ∈ G = C 4 = { e, r, r 2 , r 3 } acts on ( θ, x ) by g ( θ, x ) = ( θ, gx ) through permuting the index x : x 1 = rx 0 ; x 2 = rx 1 ; x 3 = rx 2 ; x 0 = rx 3 . Let r k ∈ G acts on vector y by rotation ρ Y ( g ) = ( cos g -sin g sin g cos g ) where g = kπ/ 2 .

̸

We construct function f in the following way: for each x ∈ X , choose l x : R → R 2 and define f ( θ, x ) = l x ( θ ) . Notice that when l gx = ρ Y ( g ) l x ( θ ) , f is G -equivariant. We define l x ( θ ) = ( p x ( θ ) , q x ( θ )) where p x and q x are cubic polynomials of x , i.g., p x with coefficients a, b, c, d will be p x = ax 3 + bx 2 + cx + d . We choose p x and q x with different coefficients for each x such that f is not equivariant, i.e., l gx = ρ Y ( g ) l x ( θ ) . For each run, we generate a function f , sample data θ, x , and evaluate the data obtaining y . Then we train neural networks using L 2 loss till converge. Eventually, we sample another set of data to evaluate the empirical L 2 error as well as the theoretical L 2 error.

I.5 Robotic Experiment

In the robotic manipulation experiment, the state s is defined as a top-down RGBD image of the workspace centered at the gripper's position (Figure 15 middle). The action a = ( x, y, z, θ, λ ) is defined as the change of position ( x, y, z ) and top-down orientation ( θ ) of the gripper, with the

gripper open width ( λ ). For a D 1 = { 1 , g } group where g represents a horizontal flip, the group action on the state space gs is defined as flipping the image; the group action on the action space ga is defined as flipping the y and θ action and leaving the other action components unchanged, ga = ( x, -y, z, -θ, λ ) . We define a D 1 -equivariant policy network π : s ↦→ a using e2cnn [60], where the output action of π will flip accordingly when the input image is flipped (Figure 15 bottom). We train the network using the Adam [24] optimizer with a learning rate of 10 -3 and weight decay of 10 -5 . For each run, we train the network for a total of 20k training steps, where we perform evaluation for 100 episodes every 2k training steps. We report the highest success rate of the 10 evaluations as the result of the run.

We develop the experimental environments in the PyBullet [12] simulator, based on the BullatArm benchmark [57]. In the Stacking (correct equivariance) and Sorting (incorrect equivariance) experiment, we gather a total of 400 episodes of demonstrations, where 400 c of them are Stacking and the rest 400(1 -c ) are Sorting. In evaluation, the task follows the same distribution, where 100 c of the evaluation episodes are Stacking and the rest are Sorting. Notice that the agent can distinguish the Stacking and Sorting tasks because the object colors are different for the two tasks (green and blue for stacking, yellow and red for sorting). In the Sorting (extrinsic equivariance) experiment, we also use 400 episodes of demonstrations.

Specifically, in Sorting, the cube and the triangle are initially placed randomly, within a distance of ± 1 . 5 cm from the horizontal mid-line of the workspace. The objective is to push the triangle at least 9 cm toward left and to push the cube at least 9 cm toward right, while ensuring that both objects remain within the boundaries of workspace. In Stacking, two blocks are randomly initialized on the floor of the workspace. The goal is to pick up the triangle and place it on top of the cube. The workspace has a size of 30 cm × 30 cm × 25 cm .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors analyze how equivariant models behave under mismatches between the symmetries of the model and data distribution.  They advance three notions, correct, extrinsic, and incorrect pointwise equivariance.  They then propose bounds on the error for equivariant neural networks that are sensitive to the data distribution and give an example where having an equivariant model can hurt learning performance.\n\n(Edit, 8/13: in light of comments by the Authors, I have changed my score to a 5)\n\nSTRENGTHS:\nTheoretically, the bounds present are more general than those proposed in Wang et al. 52.  The examples clearly demonstrate the underlying mathematical principles involved.  Moreover the error bounds presented, while not unsurprising to practitioners more versed in group theory, seem mathematically solid.  On the whole, the task of attempting to precisely characterize how the data distribution interacts with symmetries imposed on the model is a worthy one.\n\nWEAKNESSES:\nWhile the paper is solid, I think certain difficulties in presentation prevent it from fulfilling its potential.  Most immediately, I think there are some issues in the presentation in the concepts of pointwise correct / incorrect / extrinsic equivariance.  First, equivariance is a property of maps rather than spaces: saying a point has correct / incorrect / extrinsic equivariance, or even equivariance of any kind, isn’t precise.  It would be better to say something the lines of “h has correct / incorrect / extrinsic equivariance with respect to f and p at x”, particularly since the notion of correctness here is dependent on the model: this has the advantage of being a more close parallel with Defs 3.1-3.3. Second (forgive me if I missed it), but at no point is the notion of pointwise equivariance being dependent on specific group elements used in the rest of the paper.  It is not clear what is gained by this added generality.\n\nContinuing on to Section 5, it is not clear to me to what extent the notions advanced in section 4 are necessary for stating these bounds: if I understand correctly the bounds presented are for general probability distributions and are independent of the definitions proposed in 4.1-4.3.\n\nSection 6 seems a bit trivial for me: it seems obvious to me that if you can’t represent a decision boundary due to a constraint in the model you might end up having a bad day.  Indeed, the specific example shown here seems to follow straightforwardly from https://doi.org/10.48550/arXiv.2110.07472.  I think this section should be moved up and be a motivating example for the bounds in Section 5.  \n\nI feel like the meat of the paper is the error bounds, more specifically Theorem 5.8.  I’m not convinced the concepts in Definitions 4.1-4.3 contribute much towards the construction of this bound.  I would recommend a rewrite of the paper that focuses more on the bounds, their implications and applications, and maybe expand further.\n\nFinally, (and this might just be me being a pedant), but I’m not sure the title really is reflective of the content of the paper.  It is not clear to me what the general theory is that is being advanced.  Typically a “general theory” is a precise characterization of a class of mathematical objects.  If this is intended to refer to the decomposition into correct / incorrect / extrinsic equivariance, then this characterization seems to reduce to the fact that that equalities can be true, not true, or involve the empty set.  I don’t think any insight is gained innately from this characterization, rather than this being a useful language to describe further results in the text.  The error bounds, while good contributions, are specific to certain choices of loss function.  I mention this not to diminish their contributions, merely to observe that they are not “general” in a mathematical sense.\n\n\n(edited for some spelling mistakes, although I'm sure more of them abound...)\n\nQUESTIONS:\nI’m having a little trouble parsing the notion of Fundamental domain in Definition 5.1.  What happens if the orbit of the passes over X multiple times.  For instance, consider the real line, where the group is S_8 and the group action is the antisymmetric representation (reflect the line on odd permutations)?\n\nLIMITATIONS:\nThe limitations of the work have been appropriately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a general theory of correct, incorrect, and extrinsic equivariance of functions, mainly extending the framework of Wang et al., 2023 to more general case of pointwise equivariance of functions defined for pairs of group element and input data. The theory mainly concerns deriving lower bound of errors for classification and (invariant and equivariant) regression tasks. The authors further identify cases where extrinsic equivariance can be harmful for performance opposed to the empirical observations of Wang et al., 2023. The authors provide a range of experiments mainly on empirically verifying the derived lower bound of errors, and also demonstrating the cases where certain extrinsic equivariances can be harmful.\n\nWang et al., The surprising effectiveness of equivariant models in domains with latent symmetry (2023)\n\nSTRENGTHS:\nS1. The theory proposed in the paper is indeed quite general as it considers pointwise equivariance, and covers a large range of partial symmetries. The fact that it addresses a wide range of tasks (classification, invariant regression, equivariant regression) also strengthens the generality of the paper.\n\nS2. The paper is overall well written, with intuitive illustrations on pointwise equivariance and experimental setups, as well as results.\n\nS3. The experimental results support the main claims of the paper on error bounds and harmful cases of extrinsic equivariances.\n\nWEAKNESSES:\nW1. A weakness of this work is that, while certain cases are presented where extrinsic equivariances can be harmful, it offers little principled understanding or general theory of in which specific cases extrinsic equivariance is harmful or beneficial.\n\nW2. While the presented theory on pointwise equivariance covers a wide range of approximate or misspecified symmetries, I am not sure if it is immediately useful, since in many applications involving approximately or misspecified equivariant neural networks, we would not be able to exactly know the extent of correct, incorrect, and extrinsic pointwise equivariance of the model (unlike the synthetic experimental setups considered in the paper). This could be a weakness of the proposed theoretical framework.\n\nQUESTIONS:\nI have no particular questions, but would like to hear the opinions of the authors on the aforementioned weaknesses.\n\nLIMITATIONS:\nThe authors have identified the limitation of the work in Section 8.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAn obvious limitation of equivariant networks is the assumption that the symmetry they hard-code matches the symmetry of the underlying ground truth function exactly. What happens when the symmetry is only partially present in the domain affecting a symmetry mismatch between the ground-truth function and the neural network? This paper presents some systematic analysis of this situation building on prior work by Wang et al. 2023.\n\nWang et al. classified models as having \"correct\" equivariance (the underlying function provided by nature matches the symmetry of the equivariant model), \"incorrect\" equivariance (there is a disagreement between the underlying true function and that encoded by the equivariant network), and \"extrinsic\" equivariance (model symmetry transforms in-distribution data to out-of-distribution data). This paper defines these notions pointwise which allows one to generalize to a continuum of equivariance types when some tasks may have different proportions of these three situations (and does not exclusively conform to only one of them). Lower bounds are provided on the model error from incorrect model symmetry (while improving some earlier results) for classification. Morally similar lower bounds are provided for the regression case in terms of the variance of the function to be modelled over the orbit of the group under consideration. It is also shown that extrinsic equivariance can be harmful (which doesn't contradict but can still be contrasted with earlier experimental results in the literature).\n\nThe theoretical results are provided in section 5 (after first introducing the notions of correct, incorrect and equivariance of Wang et al, and then stating their pointwise generalizations which open the path to interpolate between them -- a central aim of the paper). The lower bound for classification is intuitive and basically says that it is equal to the integral of the total dissent over the fundamental domain. The total dissent measures how many elements in the orbit of G have a different label than the majority label. Two examples are given for binary and multiclass cases to build intuition (also illustrated in figures 3 and 4). Next, the case of invariant regression is treated. The result is similar in spirit to the result from earlier -- the error is bounded by an integral over the fundamental domain of p(Gx) times the variance of the function on the orbit Gx (so instead of the dissent, we look at the variance). A slightly more careful consideration permits an easy generalization for equivariant regression. A simple theoretical argument is then provided to show that extrinsic can be harmful for generalization, also raising the question that it remains open to understanding its effect on generalization.\n\nSTRENGTHS:\n- The paper considers an important problem that has only recently started getting treated in the literature. It is clear that equivariant networks make a strong assumption about the symmetry of the ground truth function. However, it is not clear how much a mismatch matters. This paper provides some approximation lower bounds due to the mismatch for some different settings. \n-  Being able to interpolate between the three types can definitely help improve model generalization and permit more flexible models.\n- While many of the points raised in the paper seem somewhat obvious at times, and computing the lower bound might not be possible in practice, the presented work can still give some guidance on model selection.\n\nWEAKNESSES:\n- Somewhat fortuitously, I was a reviewer for this paper for an earlier iteration. Whatever weaknesses that I had raised at the point seem to have been adequately addressed by the authors (in fact they were right then). I also notice that the other weaknesses bought up by other reviewers at the time (mostly in terms of experiments, and comparison to the work of Wang) have also mostly been incorporated. While I can nitpick, I have no hesitation in simply voting for acceptance.\n\nQUESTIONS:\nNo\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
The paper analyzes error bounds for models constrained to satisfy symmetries that only partially agree with the ground truth functions. The main suggestion is to generalize previous definitions, to the point level, allowing to derive a lower bound on the model error with respect to the volume of the portion of the domain where symmetries are mismatched. The theoretical analysis is validated on some toy examples and some (relatively) small-scale datasets

STRENGTHS:
The proposed analysis including the definitions, propositions, and theorems seems to be simple.

The paper is well-written and easy to follow. I appreciate the illustrations and toy examples provided. 

The experiments seem to support the theoretical analysis. Where mismatches occur, reasonable explanations are provided.

WEAKNESSES:
Missing discussion

I feel the text should elaborate more on the assumptions taken in the analysis of error bound. For example, it is assumed that the model assigns labels by taking a majority vote in an orbit. How reasonable is this assumption in relation to existing equivariant models? 

In addition, it would be interesting to incorporate a model that approximates the majority vote by sampling for one of the experiments.  E.g., does it improve the digit classification model?

QUESTIONS:
No further questions.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an extension of a lower bound on error in finite labeled classification introduced in \"The Surprising Effectiveness of Equivariant Models in Domains with Latent Symmetry\" and investigates lower bounds on error for G-equivariant and G-invariant regression, offering valuable insights into the performance of symmetry-preserving models in regression tasks.\n\nSTRENGTHS:\nThis extension holds significant practical importance, particularly when considering real-world natural symmetry groups such as $G = SO(3)$, which go beyond the scope of the original paper. Moreover, the authors have done an excellent job in presenting the material in an accessible manner, making the paper easy to read and well-structured.\n\nWEAKNESSES:\n1.\nThe paper's focus on pointwise definitions rather than emphasizing their implications may obscure the novelty of the results, potentially hindering a clear understanding of the significance of their contributions.\n\n2.\nWhile the experiments conducted in the paper offer valuable insights, their omission of infinite groups may limit the demonstration of the full strength of the new lower bound. Additionally, the observations suggesting equivariant models' potential ineffectiveness in classical learning tasks raise valid questions about their relevance and applicability.\n\n3.\nAlthough the paper draws upon certain notions from \"The Surprising Effectiveness of Equivariant Models in Domains with Latent Symmetry,\" it does so only partially. This partial utilization of concepts, coupled with the use of different evaluation metrics such as \"accuracy\" in the original paper and \"error\" in this work, may create challenges when attempting to directly compare the results between the two studies.\n\nQUESTIONS:\n1. \nCan you elaborate on the third experiment? Which state is flipped? (line 532, page 9)\n\n2.\nDid you mean \"orbits\" instead of conjugates in the definition of the fundamental domain? (line 158, page 4)\n\nLIMITATIONS:\nThe author did address the limitations of reflection symmetries on the expressivity of models in Chapter 6 (line 254, page 7), and the requirement of knowledge about the density function (line 386, page 9)", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work addresses the situation in which data and model equivariance does not match exactly. It extends previous work from Wang et al., by proposing a pointwise version of their definition of correct, incorrect and extrinsic equivariance. The usefulness of these new concepts are demonstrated in three new performance lower-bounds derived from classification and regression problems. A series of examples and experiments are presented to illustrate the new proposed notions, confirm the lower-bounds derived and show that they usually seem tight in practice.\n\nSTRENGTHS:\n### Originality\nAlthough it builds heavily on previous work from Wang [52], the paper presents novel theoretical concepts and results, as well as new experiments supporting them.\n\n### Clarity\nOverall, the paper is well written and structured.\n\n### Quality\nThe paper presents new theoretical results with their corresponding proofs, which look sound to be (disclaimer: I have some familiarity with group theory, but I am not an expert). It also presents interesting results in a large range of experiments. The latter are all rather small/toy, but are still quite convincing IMO.\n\nWEAKNESSES:\n### Clarity\n\nMinor:\n\n- I think there is a mistake in the xlabel of fig 7b. Shouldn't it be \"incorrect - correct\"? Same for figure 7a: should be \"correct - extrinsic\" I guess, since when x=1 you have the highest INV model performance, which should correspond to c=1 and e=0 according to the text.\n- L.363: I think you wanted to reference fig.9, not 8.\n\n### Clarity and Quality\n\n1. I think the definition of $p$ could be clarified. It is introduced in line 92 as the “probability density function of the domain”, so, at first reading, I thought p(x) was the “true” underlying population distribution from which both training and test examples are samples. This seems to be confirmed by line 387 in the conclusion: “our theoretical lower bounds require domain knowledge like the density function over the domain”.  But in section 6, it seems that you define extrinsic equivariance with respect to the actual examples in the training set, which is not the same thing: “F_E corresponds to an extrinsically equivariant class for **this data**” l.272\n\n2. I also have other few questions regarding the example presented in section 6. First of all, does the data from figure 5 represent training or test data? Is the data $S$ the whole support of $p(x)$? What is the true labeling function $f$ in this example?\nThese elements seem important to conclude. The reason I ask is that if $f$ is indeed the “exclusive or” on (x,y) coordinates (i.e. it is indeed C2-invariant) and if the examples from figure 5a are just the training data but the test data can go outside (e.g. their symmetric elements in fig 5b could be the test set), then despite the 0% error rate of the unconstrained linear model on the training data, it would learn an incorrect labeling function and its test performance would be 50%, while the invariant model would still have a 25% error rate, which would be better.\n\n3. If $p$ really denotes the population distribution, I wonder whether we can talk about $p$ independently of $f$ and vice-versa. From your figures 1 and 2, it seems that $f$ can be defined outside of the support of $p(x)$, and I wonder whether this makes sense. For example, if $f$ is the labeling function in a digit classification problem, what should be its output for an image of the digit “9” rotated by 90 degrees? Should it be 9 or 6?\n\n### Originality:\n\nMinor:\n\n4. In the related work, some references to class-specific and instance-specific automatic data augmentation works are missing ([1, 2, 3] for example), while they are strongly related to the idea of pointwise invariance proposed in the paper.\n\n[1] https://arxiv.org/abs/2106.13695\n\n[2] https://arxiv.org/abs/1510.02795\n\n[3] https://arxiv.org/abs/2206.00051\n\nQUESTIONS:\nSee 3 questions in the weaknesses section above.\n\nLIMITATIONS:\nYes, the authors mention one limitation of their work, which is the need to know $p$, which is never the case in practice.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}