Title: Approximation-Generalization Trade-offs under (Approximate) Group Equivariance
Venue: NeurIPS 2023
Accepted: yes
Keywords: Equivariance

Invariance

Generalization

Equivariant Neural Networks

Approximation Error
Abstract: The explicit incorporation of task-specific inductive biases through symmetry has emerged as a general design precept in the development of high-performance machine learning models. For example, group equivariant neural networks have demonstrated impressive performance across various domains and applications such as protein and drug design. A prevalent intuition about such models is that the integration of relevant symmetry results in enhanced generalization. Moreover, it is posited that when the data and/or the model exhibits only approximate or partial symmetry, the optimal or best-performing model is one where the model symmetry aligns with the data symmetry. In this paper, we conduct a formal unified investigation of these intuitions. To begin, we present quantitative bounds that demonstrate how models capturing task-specific symmetries lead to improved generalization. Utilizing this quantification, we examine the more general question of dealing with approximate/partial symmetries. We establish, for a given symmetry group, a quantitative comparison between the approximate equivariance of the model and that of the data distribution, precisely connecting model equivariance error and data equivariance error. Our result delineates the conditions under which the model equivariance error is optimal, thereby yielding the best-performing model for the given task and data.

Approximation-Generalization Trade-offs under (Approximate) Group Equivariance

Abstract

The explicit incorporation of task-specific inductive biases through symmetry has emerged as a general design precept in the development of high-performance machine learning models. For example, group equivariant neural networks have demonstrated impressive performance across various domains and applications such as protein and drug design. A prevalent intuition about such models is that the integration of relevant symmetry results in enhanced generalization. Moreover, it is posited that when the data and/or the model may only exhibit approximate or partial symmetry, the optimal or best-performing model is one where the model symmetry aligns with the data symmetry. In this paper, we conduct a formal unified investigation of these intuitions. To begin, we present general quantitative bounds that demonstrate how models capturing task-specific symmetries lead to improved generalization. In fact, our results do not require the transformations to be finite or even form a group and can work with partial or approximate equivariance. Utilizing this quantification, we examine the more general question of model mis-specification i.e. when the model symmetries don't align with the data symmetries. We establish, for a given symmetry group, a quantitative comparison between the approximate/partial equivariance of the model and that of the data distribution, precisely connecting model equivariance error and data equivariance error. Our result delineates conditions under which the model equivariance error is optimal, thereby yielding the best-performing model for the given task and data. Our results are the most general results of their type in the literature.

1 Introduction

It is a common intuition that machine learning models that explicitly incorporate task-specific symmetries tend to exhibit superior generalization. The rationale is simple: if aspects of the task remain unchanged or change predictably when subject to certain transformations, then, a model without knowledge of them would require additional training samples to learn to disregard them. While the general idea is by no means new in machine learning, it has experienced a revival in interest due to the remarkable successes of group equivariant convolutional neural networks (GCNNs) [14, 38, 29, 16, 13, 60, 8, 34, 24, 66]. Such networks hard-code equivariance to the action of an algebraic group on the inputs and have shown promise in a variety of domains [55, 5, 42, 2, 28, 62, 25, 52, 18, 61, 68], in particular in those where data is scarce, but exhibits clear symmetries. Further, the benefits of encoding symmetries in this manner extend beyond merely enhancing generalization. Indeed, in

numerous applications in the physical sciences, neglecting to do so could lead to models producing unphysical outputs. However, the primary focus of most of the work in the area has been on estimating functions that are strictly equivariant under a group action. It has often been noted that this may impose an overly stringent constraint as real-world data seldom exhibits perfect symmetry. The focus on strict equivariance is partly due to convenience: the well-developed machinery of group representation theory and noncommutative harmonic analysis can easily be marshalled to design models that are exactly equivariant. A general prescriptive theory for such models also exists [29, 16]. In a certain sense, the basic architectural considerations of such models are fully characterized.

Recent research has begun to explore models that enforce only partial or approximate equivariance [40, 59, 23, 56]. Some of these works suggest interpolating between models that are exactly equivariant and those that are fully flexible , depending on the task and data. The motivation here is analogous to what we stated at the onset. If the data has a certain symmetry - whether exact, approximate or partial - and the model's symmetry does not match with it, its performance will suffer. We would expect that a model will perform best when its symmetry is correctly specified, that is, it aligns with the data symmetry.

Despite increased research interest in the area, the theoretical understanding of these intuitions remains unsatisfactory. Some recent work has started to study improved generalization for exactly equivariant/invariant models (see section 2). However, for the more general case when the data and/or model symmetries are only approximate or partial, a general treatment is completely missing. In this paper, we take a step towards addressing this gap. First, we show that a model enforcing task-specific invariance/equivariance exhibits better generalization. Our theoretical results subsume many existing results of a similar nature. Then we consider the question of model mis-specification under partial/approximate symmetries of the model and the data and prove a general result that formalizes and validates the intuition we articulated above.

We summarize below the main contributions of the paper:

We present quantitative bounds, the most general to date, showing that machine learning models enforcing task-pertinent symmetries in an equivariant manner afford better generalization. In fact, our results do not require the set of transformations to be finite or even be a group. While our investigation was initially motivated by GCNNs, our results are not specific to them.

Using the above quantification, we examine the more general setting when the data and/or model symmetries are approximate or partial and the model might be misspecified relative to the data symmetry. We rigorously formulate the relationship between model equivariance error and data equivariance error, teasing out the precise conditions when the model equivariance error is optimal, that is, provides the best generalization for given data. To the best of our knowledge, this is the most general result of its type in the literature.

2 Related Work

As noted earlier, the use of symmetry to encode inductive biases is not a new idea in machine learning and is not particular to neural networks. One of the earliest explorations of this idea in the context of neural networks appears to be [46]. Interestingly, [46] originated in response to the group invariance theorems in Minsky & Papert's Perceptrons [36], and was the first paper in what later became a research program carried out by Shawe-Taylor & Wood, building more general symmetries into neural networks [63, 48, 49, 65, 64], which also included PAC style analysis [50]. While Shawe-Taylor & Wood came from a different perspective and already covered discrete groups [14, 38], modern GCNNs go beyond their work [29, 16, 13, 60, 8, 34, 24]. Besides, GCNNs [14] were originally proposed as generalizing the idea of the classical convolutional network [32], and seem to have been inspired by symmetry considerations from physics [15]. Outside neural networks, invariant/equivariant methods have also been proposed in the context of support vector machines [43, 57], general kernel methods [39, 26], and polynomial-based feature spaces [44].

On the theoretical side, the assertion that invariances enhance generalization can be found in many works, going back to [47]. It was argued by [1] that restricting a classifier to be invariant can not increase its VC dimension. Other examples include [3, 4, 51, 41]. The argument in these, particularly [51, 41] first characterizes the input space and then proceeds to demonstrate that certain data transformations shrink the input space for invariant models, simplifying the input and improving generalization. Some of our results generalize their results to the equivariant case and for more general transformations.

Taking a somewhat different tack, [22] showed a strict generalization benefit for equivariant linear models, showing that the generalization gap between a least squares model and its equivariant counterpart depends on the dimension of the space of anti-symmetric linear maps. This result was adapted to the kernel setting in [19]. [9] studied sample complexity benefits of invariance in a kernel ridge regression setting under a geometric stability assumption, and briefly discussed going beyond group equivariance. [53] expands upon [9] by characterizing minimax optimal rates for the convergence of population risk in a similar setting when the data resides on certain manifolds. [35] characterizes the benefits of invariance in overparameterized linear models, working with invariant random features projected onto high-dimensional polynomials. Benefits of related notions like data augmentation and invariant averaging are formally shown in [33, 12]. Using a standard method from the PAC-Bayesian literature [37] and working on the intertwiner space, [7] provide a PAC-Bayesian bound for equivariant neural networks. Some improved generalization bounds for transformation-invariant functions are proposed in [67], using the notion of an induced covering number. A work somewhat closer to some of our contributions in this paper is [20], which gives PAC bounds for (exact) group equivariant models. However, we note that the proof of the main theorem in [20] has an error (see the Appendix for a discussion), although the error seems to be corrected in the author's dissertation [21]. An analysis that somewhat overlaps [20], but goes beyond simply providing worst-case bounds was carried out by [45].

Finally, recent empirical work on approximate and partial equivariance includes [40, 59, 23, 56]. These make the case that enforcing strict equivariance, if misspecified, can harm performance, arguing for more flexible models that can learn the suitable degree of equivariance from data. However, there is no theoretical work that formalizes the underlying intuition.

3 Preliminaries

In this section, we introduce basic notation and formalize some key notions that will be crucial to ground the rest of our exposition.

3.1 Learning with equivariant models

Learning problem. To begin, we first describe the general learning problem which we adapt to the equivariant case. Let's say we have an input space X and an output space Y , and assume that the pairs Z = ( X,Y ) ∈ X × Y are random variables having distribution D . Suppose we observe a sequence of n i.i.d pairs Z i = ( X i , Y i ) ∼ D , and want to learn a function ˜ f : X → Y that predicts Y given some X . We will consider that ˜ f belongs to a class of functions ˜ F ⊂ { ˜ f : X → Y} and that we work with a loss function /lscript : X × Y → [0 , ∞ ). To fully specify the learning problem we also need a loss class , comprising of the functions f ( x, y ) := /lscript ( ˜ f ( x ) , y ) : X × Y → [0 , ∞ ). Using these notions, we can define:

The quantities Pf and P n f are known as the risk and the empirical risk associated with ˜ f , respectively. In practice we have access to P n f , and we estimate Pf by it. One of the chief quantities of interest in such a setup is the worst-case error of this approximation on a sample { Z i } = { Z i } n i =1 , that is, the generalization error:

Group actions. We are interested in the setting where a group G acts by measurable bijective transformations over X and Y , which can also be seen as a G -action over Z , denoted

by g · Z = ( g · X,g · Y ), where g ∈ G . Such actions can also be applied to points ( x, ˜ f ( x )) in the graph of functions ˜ f , transforming any ˜ f ∈ ˜ F into g · ˜ f : x ↦→ g -1 · ˜ f ( g · x ). The set G · ˜ f := { g · ˜ f : g ∈ G } forms the orbit of ˜ f under the action of G . Note that although we treat G as an abstract group throughout, we will sometimes abuse notation and use the same letter to also refer to the representations of G as transformations over the spaces in which X,Y,Z, ˜ f, f live.

Equivariant models. We are now in a position to specify the learning setup with equivariant models. We say that ˜ f ∈ ˜ F is equivariant, if G · ˜ f = { ˜ f } . Further, we say that the hypothesis class is G -equivariant if all ˜ f ∈ ˜ F are equivariant. Note that if the G -action over Y is trivial, that is, if we have g · Y = Y for all g ∈ G,y ∈ Y ), then ˜ f being equivariant reduces to requiring ˜ f ( g · x ) = ˜ f ( x ) for all g, x . In such a case the model ˜ f is said to be invariant . Also note that ˜ f ( x ) being equivariant corresponds to f ( x, y ) = /lscript ( ˜ f ( x ) , y ) being invariant under some suitable product G -action over X × Y .

Averaging over a set of transformations. Let G be a set of transformations of Z . The G -averaged generalization error for arbitrary F by taking averages E g with respect to the natural probability measure P G 3 on G is as follows:

Note that the above equals GenErr ( F , { Z i } , D ) if F is composed of invariant functions.

Data augmentation/Orbit averaging. Following our discussion above, it might be instructive to consider the case of data augmentation. Note that a data distribution which is invariant under some G -action can be created from any D by averaging over G -orbits: we take g to be a random variable uniformly distributed over a compact set of transformations G . Then let D ( G ) := 1 | G | ´ G g · D . This is the distribution of the dataset obtained from D under G -augmentation. Then we have by change of variables:

Note that the notions of orbit and averaging used above do not rely on the property that G is a group in order to be well-defined.

3.2 Partial and Approximate Equivariance

Since our analysis treats the general case where the model and/or data symmetries are not strict, we now state some basic attendant notions.

Error to equivariance function. We first need a measure to quantify how far off we are from perfect equivariance. To do so, we can define a function in terms of f ( x, y ) as:

The crucial property here is that ee ( f, g ) = 0 iff g · f = f . Note that one could use other distances or discrepancies (e.g. like the /lscript 2 norm) directly in terms of ˜ f , using which we can measure the error between g · ˜ f and ˜ f .

More general sets of transformations. We can use ee ( · ) to precisely specify the notions of partial and approximate equivariance that we work with. We consider the general case where the underlying transformations on ( X,Y ) still have a common label set G and are assumed to be bijective over X and Y , but don't necessarily form a group. For /epsilon1 > 0 and f ∈ F we consider the /epsilon1 -stabilizer

It is fairly easy to see that this subset is automatically a group for /epsilon1 = 0, but most often will not be when /epsilon1 > 0. The latter corresponds to partial symmetry and thus to the setting

of partial equivariance . On the other hand, if we consider specific hypotheses such that for some /epsilon1 > 0 we have Stab /epsilon1 ( f ) = G for all f ∈ F , then we are working in the setting of approximate equivariance [59, 56].

Conditions on G . We conclude this section with a remark on the nature of G we consider in our paper. In our bounds, we first restrict to finite G . However, this setting can be extended to compact groups which are doubling with respect to a metric that respects the composition operation. That is, a distance d G over G , such that d G ( gh, g ′ h ) = d ( g, g ′ ). The main examples we have in mind are compact Lie groups such as S 1 , SO ( n ) , O ( n ), and their finite subgroups like Z /N Z , as well as their quotients and products.

4 PAC bounds for G -equivariant hypotheses

In this section, we study the generalization properties of equivariant models. More specifically, we derive PAC bounds for both exact and approximately G-equivariant hypotheses. But first, we present some preliminaries related to PAC bounds in what follows.

PAC learning bounds.

We follow the standard approach (such as that outlined in [10]). We start with the following concentration bound, where F is composed of functions with range [ -M/ 2 , M/ 2]:

where R n ( F ) denotes the Rademacher complexity, which is given by

and where Z 1 , . . . , Z n ∼ D are n i.i.d. samples and σ = ( σ 1 , . . . , σ n ) denotes the so-called Rademacher variable, which is a uniformly distributed random variable over {-1 , 1 } n , independent of the Z i 's. R ( F Z ) can be bounded using a classical technique for bounding the expected suprema of random processes indexed by the elements of a metric space, variously called the Dudley entropy integral or the chaining bound. The result below was proven in [45, Lemma 3] (also see slightly stronger results in [58, Thm. 17] or [6, Thm. 1.1])

Recall that for a metric space ( X, d X ), the covering number N ( X,/epsilon1, d X ) is the smallest number of balls of radius /epsilon1 required to cover X . In (4.2) we use the cover numbers of F in supremum distance, i.e. the distance between two functions f, g ∈ F , ‖ f -g ‖ ∞ := sup z ∈Z | f ( z ) -g ( z ) | ). It is known that by rescaling the fundamental estimate due to Kolmogorov-Tikhomirov [54, eq. 238, with s = 1, and eq. 1], and under the mild assumption that F is composed of 1-Lipschitz 4 functions on Z with values in an interval [ -M/ 2 , M/ 2], for a centralizable 5 metric space Z , the following holds

Before we start stating our results, we need to define a few more notions:

Doubling and Hausdorff dimensions. If in a metric space ( X, d X ), every ball of radius R can be covered by at most λ balls of radius R/ 2, the space has doubling dimension ddim ( X ) = log 2 λ . The doubling dimension coincides with the usual notion of (Hausdorff) dimension dim X , i.e. ddim X = dim X , in case X is a compact manifold with bounded injectivity radius, in particular it equals d if X ⊂ R D is a regular d -dimensional submanifold or if D = d and X is a regular open subset such as a ball or a cube.

/negationslash

Discrete metric spaces. A metric space ( X, d X ) is discrete if it has strictly positive minimum separation distance δ X := min x = x ′ ∈ X d X ( x, x ′ ) > 0. We then have

which is especially relevant for finite groups G endowed with the word distance (i.e. d G ( g, g ′ ), which is the minimum number of generators required to express g -1 g ′ ), for which δ G = 1. Now, note that as a straightforward consequence of the definition, there exists a universal c > 0 such that[30, 31]:

and by (4.3) we get the simplified bound (where implicit constants are universal):

By all the above considerations, we can bound the generalization error as follows:

Proposition 4.1. Assume that d = ddim ( Z ) > 2 , and 0 < δ < 1 / 2 , and let D := diam ( Z ) . Then for any probability distribution D of data over Z , with notation as in the beginning of Section 4, the following holds with probability at least 1 -δ :

the implicit constant is independent of δ, n ; only depending on Z through the constants in (4.5) .

Due to space constraints, all our proofs are relegated to the Appendix. With the above background, we now first show generalization bounds under the more general notions of equivariance we have discussed.

4.1 Generalization bounds improvement under partial or approximate equivariance

In this section, we prove generalization error bounds with the notations and definitions from Section 3.2. We consider the following sets of transformations:

We note that the strict equivariance case is recovered if, for /epsilon1 = 0, we have Stab 0 = G .

Proposition 4.2. Let Stab /epsilon1 be as above, and assume that | Stab /epsilon1 | > 0 . let Z 0 /epsilon1 ⊂ Z be a measurable choice of Stab /epsilon1 -orbit representatives for points in Z , and let ι 0 /epsilon1 : Z → Z 0 /epsilon1 be the measurable map that associates to each z ∈ Z its representative in Z 0 /epsilon1 . Let F 0 /epsilon1 := { f | Z 0 /epsilon1 : f ∈ F} and let ι 0 /epsilon1 ( D ) represent the image measure of D . Then for each n ∈ N , if { Z i } n i =1 are i.i.d. samples with Z i ∼ D and Z 0 i,/epsilon1 := ι 0 /epsilon1 ◦ Z i , then we have

The above result says that the generalization error of our setting of interest could roughly be obtained by working with a reduced set of only the orbit representatives for points in Z . This is in line with prior work such as [51, 20, 41]. However, note that our result is already more general and does not require that the set of transformations form a group. With this, we now need an understanding of the covering of spaces of representatives Z 0 /epsilon1 and the effect of Stab /epsilon1 on them. The answer is evident in case Stab /epsilon1 = G , that G , Z 0 are manifolds, and Z = Z 0 × G . Since ddim coincides with topological dimension, and we immediately have

Intuitively, the dimensionality of G can be understood as eliminating degrees of freedom from Z , and it is this effect that improves generalization by n -1 / ( d -dim ( G )) -n -1 /d .

We now state a simplified form of Theorem A.3, which is sufficient for proceeding further. Our results generalize [51, Thm. 3]: we allow for non-product structure, possibly infinite sets of transformations that may not form a group, and we relax the distance deformation hypotheses for the action on Z ).

Corollary 4.3 (of Thm. A.3) . With the same notation as above, assume that for Stab /epsilon1 the transformations corresponding to the action of elements of Stab /epsilon1 satisfy the following for some L > 0 , and for a choice of a set of representatives Z 0 /epsilon1 ⊂ Z of representatives of Stab /epsilon1 -orbits:

For all z 0 , z ′ 0 ∈ Z 0 /epsilon1 and all g ∈ Stab /epsilon1 it holds that d ( z 0 , z ′ 0 ) ≤ L d ( g · z 0 , g · z ′ 0 ) .

For all g, g ′ ∈ Stab /epsilon1 it holds that d G ( g, g ′ ) ≤ L dist ( g · Z 0 /epsilon1 , g ′ · Z 0 /epsilon1 ) 6 .

Then for each δ > 0 there holds N ( Z 0 /epsilon1 , δ ) ≤ N ( Z , δ/ 2 L ) / N ( Stab /epsilon1 , δ ) .

To be able to quantify the precise generalization benefit we need a bound on the quantity N ( Stab /epsilon1 , δ ). For doing so, we assume that Stab /epsilon1 is a finite subset of a discrete group G , or is a positive measure subset of a compact group G . As before, let | G | and d G denote /sharpG and ddim ( G ) respectively for finite G . Also, denote the Hausdorff measure and dimension by Vol ( G ) and dim ( G ) for compact metric groups. Note that the minimum separation δ G is zero for dim ( G ) > 0. Then our covering bound can be expressed in the following umbrella statement:

In order to compare the above to the precise equivariance case, we later use the factor

which measures the richness of Stab /epsilon1 compared to the whole ambient group G , in terms of error /epsilon1 . The fact that Dens ( /epsilon1 ) > 0 follows from our assumption that | Stab /epsilon1 | > 0. Furthermore, Dens ( /epsilon1 ) is always an increasing function of /epsilon1 , as follows from the definition of Stab /epsilon1 . With these considerations on coverings, we can state a general result quantifying the generalization benefit, where if we set /epsilon1 = 0, we recover the case of exact group equivariance.

Theorem 4.4. Let /epsilon1 > 0 be fixed. Assume that for a given ambient group G the almost stabilizers Stab /epsilon1 satisfy assumption (4.7) and that the assumptions of Corollary 4.3 hold for some finite L = L /epsilon1 > 0 . Then with the notations of Proposition 4.1 we have with probability ≥ 1 -δ

where

The interpretation of the above terms is direct: the term 2 /epsilon1 is the effect of approximate equivariance, and the term ( E /epsilon1 ) includes the dependence on | Stab /epsilon1 | and thus is relevant to the study of partial equivariance. In general the Lipschitz bound L /epsilon1 is increasing in /epsilon1 as well, since Stab /epsilon1 includes more elements of G as /epsilon1 increases, and we can expect that actions of elements generating higher equivariance error in general have higher oscillations.

5 Approximation error bounds under approximate data equivariance

In the introduction, we stated that we aim to validate the intuition that that model is best whose symmetry aligns with that of the data. So far we have only given bounds on the generalization error. Note that our bounds hold for any data distribution: in particular, the bounds are not affected by whether the data distribution is G -equivariant or not. However,

the fact that data may have fewer symmetries than the ones introduced in the model will have a controlled effect on worsening the approximation error, as described in this section.

However, controlling the approximation error has an additional actor in the fray to complicate matters: the distinguishing power of the loss function. Indeed, having a degenerate loss function with a large minimum set will not distinguish whether the model is fit to the data distribution or not. Thus, lowering the discrimination power of the loss has the same effect as increasing the function space for which we are testing approximation error.

Assuming at first that ˜ F is composed of G -equivariant functions, we compare its approximation error to the one of the set M of all measurable functions m : X → Y . Recall that f ( Z ) = /lscript ( ˜ f ( X ) , Y ) is a positive function for all ˜ f ∈ ˜ F , thus, denoting by M ′ := { F ( x, y ) = /lscript ( m ( x ) , y ) , m ∈ M} , the approximation error gap can be defined and bounded as follows (where F ∗ ∈ M ′ is a function realizing the first minimum in the second line below):

With applications to neural networks in mind, we will work with the following simplifying assumptions:

The loss function quantitatively detects whether the label is correct, i.e. we have /lscript ( y, y ′ ) = 0 if y = y ′ , and /lscript ( y, y ′ ) ≥ ϕ ( d Y ( y, y ′ )) for a strictly convex ϕ with ϕ (0) = 0, where d Y is the distance on Y . This assumption seems reasonable for use in applications, where objective functions with strict convexity properties are commonly used, as they help for the convergence of optimization algorithms.

The minimization across all measurable functions ˜ F = M produces a zero-risk solution, i.e. min F ∈M ′ E [ F ( Z )] = 0 is achieved by a measurable function y ∗ : X → Y . By assumption 1, this implies that D is concentrated on the graph of y ∗ . This assumption corresponds to saying that the learning task is in principle deterministically solvable. Under this assumption, the task becomes to approximate to high accuracy the (unknown) solution y ∗ .

With the above hypotheses we have

As a slight simplification of Assumption 1 for ease of presentation, we will simply take Y = R d and /lscript ( y, y ′ ) = d Y ( y, y ′ ) 2 below. Note that Assumption 1 directly reduces to this case, as a strictly convex ϕ ( t ) with ϕ (0) = 0 as in Assumption 1 is automatically bounded below by a multiple of t 2 .

Now, with the aim to capture the mixed effects of approximate and partial equivariance, we introduce the following set of model classes. For /epsilon1 ≥ 0 , λ ∈ (0 , 1]:

/negationslash

In order to establish some intuition, note that Stab /epsilon1 ( F ) is increasing in /epsilon1 and as a consequence C λ,/epsilon1 is also increasing in /epsilon1 . Directly from the definition one finds that Stab 0 ( F ) is necessarily a subgroup of G , and thus we allow /epsilon1 > 0, which allows more general sets of symmetries than just subgroups of G . The most interesting parameter in the above definition of C /epsilon1,λ is λ , which actually bounds from below the "amount of prescribed symmetries" for our models. In the fully equivariant case /epsilon1 = 0 , λ = 1, we have the simple description C 0 , 1 = {F : F ⊆ ( M ( G ) ) ′ } , whose maximal element is ( M ( G ) ) ′ . However in general C /epsilon1,λ will not have a unique maximal element for λ < 1: this is due to the fact that two different elements F 1 = F 2 ∈ C /epsilon1,λ may have incomparable stabilizers, so that | Stab /epsilon1 ( F 1 ) \ Stab /epsilon1 ( F 2 ) | > 0 and | Stab /epsilon1 ( F 2 ) \ Stab /epsilon1 ( F 1 ) | > 0, and thus F 1 ∪ F 2 / ∈ C /epsilon1,λ . Our main result towards our approximation error bound is the following.

Proposition 5.1. Assume that ( Y , d Y ) is a compact metric space and consider /lscript ( y, y ′ ) = d Y ( y, y ′ ) 2 . Fix /epsilon1 ≥ 0 . Then for each λ ∈ (0 , 1] there exists an element F ∈ C /epsilon1,λ and a measurable selection ι 0 /epsilon1 : X → X 0 /epsilon1 of representatives of orbits of X under the action of Stab /epsilon1 ( F ) , such that if X 0 /epsilon1 , g are random variables obtained from X ∼ D by imposing X 0 /epsilon1 := ι 0 /epsilon1 ◦ X and X = g · X 0 /epsilon1 , then we have

In the above, as /epsilon1 increases, we see a decrease in the approximation gap, since in this case model classes F allow more freedom for approximating y ∗ more accurately. On the other hand, the effect of parameter λ is somewhat more subtle, and it has to do with the fact that as λ decreases, Stab /epsilon1 ( F ) for F ∈ C /epsilon1,λ can become smaller and the allowed oscillations of y ∗ ( g · X 0 /epsilon1 ) are more and more controlled.

Now to bound the approximation error above using the quantities that we have been working with, we will use the following version of an isodiametric inequality on metric groups G . As before we assume that G has Hausdorff dimension dim ( G ) > 0 which coincides with the doubling dimension, or is discrete. By | X | we denote the Hausdorff measure of X ⊆ G if dim ( G ) > 0 or the cardinality of X if G is finite. Then there exists an isodiametric constant C G depending only on G such that for all X ⊆ G it holds that

The above bound can be obtained directly from the doubling property of G , applied to a cover of X via a single ball of radius diam ( X ), which can be refined to obtain better and better approximations of | X | . We can now control the bound from Proposition 5.1 as follows:

Theorem 5.2. Assume that ( Y , d Y ) is a compact metric space and /lscript ( y, y ′ ) = d Y ( y, y ′ ) 2 . Let G be a compact Lie group or a finite group, with distance d G . Let ddim ( G ) be the doubling dimension of G , and assume that for all g ∈ G such that the action of g over X is defined, it holds that d ( g · x, g ′ · x ) ≤ L ′ d G ( g, g ′ ) . Then, there exists a constant C G depending only on G such that for all λ ∈ (0 , 1] and /epsilon1 > 0 , there exists an element F ∈ C /epsilon1,λ where for any data distribution D , y ∗ can be chosen Lipschitz with constant Lip ( y ∗ ) . We have

We note that, given our proof strategy of Proposition 5.1, we conjecture that actually the above bound is sharp up to a constant factor: min F∈C /epsilon1,λ AppErr ( F , D ) may have a lower bound comparable to the above upper bound. The goal of the above result is to formalize the property that the approximation error guarantees of F will tend to grow as the amount of imposed symmetries grow, as measured by λ .

6 Discussion: Imposing optimal equivariance

We have so far studied bounds on the generalization and the approximation errors, each of which take a very natural form in terms of properties of the space of orbit representatives (and thus the underlying G ). One of them depends on the data distribution, while the other doesn't. Using them we can thus state a result that quantifies the performance error, or gives the generalization-approximation trade-off, of a model based on the data distribution (and their respective symmetries). Define the performance error of a model class F ∈ C /epsilon1,λ over an i.i.d. sample { Z i } n i =1 , Z i ∼ D as

Combining Theorems 4.4 and 5.2, we get the following:

Theorem 6.1. Under the assumptions of Theorems 4.4 and 5.2, we have that, assuming all the function classes are restricted to functions with values in [ -M,M ] , then there exist

values C 1 , C 2 , C 3 > depending only on d 0 , (2 LD ) d / | G | , δ G and C = C G L ′ (1 + Lip ( y ∗ )) such that with probability at least 1 -δ ,

Note that the above bounds are not informative as λ → 0 for the continuous group case, which corresponds to the second option in Theorem 6.1. This can be sourced back to the form of Assumption (4.7), in which the covering number on the left is always ≥ 1, whereas in the continuous group case, the right-hand side may be arbitrarily small.

Thus the result is only relevant for λ away from 0. If we fix /epsilon1 = 0 and we optimize over λ ∈ (0 , 1] in the above bound, we find a unique minimum λ ∗ for the error bound, specifying the optimum size of Stab /epsilon1 for the corresponding case (see the Appendix for more details). This validates the intuition stated at the onset that for the best model its symmetries must align with that of the data. As an illustration, for C, C 1 , C 2 = 0 . 04 and C 3 = 0 . 01 with n = 1000000, the error trade-off is shown on the figure on the right.

7 Concluding Remarks

In this paper, we have provided general quantitative bounds showing that machine learning models with task-pertinent symmetries improve generalization. Our results do not require the symmetries to be a group and can work with partial/approximate equivariance. We also presented a general result which confirms the prevalent intuition that if the symmetry of a model is mis-specified w.r.t to the data symmetry, its performance will suffer. We now indicate some important future research directions, that correspond to limitations of our work:

( Model-specific bounds ) While in Theorem 6.1 we obtain the existence of an optimal amout λ ∗ without hopes to be sharp in such generality, we may expect that if we restrict to specific tasks, the bounds can become much tighter, and if so, the value λ ∗ of optimal amount of symmetries can become interpretable and give insights into the structure of the learning tasks.

( Tighter bounds beyond classical PAC theory ) For the sake of making the basic principles at work more transparent, we based our main result on long-known classical results. However, tighter data-dependent or probabilistic bounds would be very useful for getting a sharper value for the optimal value of symmetries λ ∗ via a stronger version of Theorem 6.1.

( Beyond controlled doubling dimension ) Our focus here is on groups G of controlled doubling dimension. This includes compact and nilpotent Lie groups, and discrete groups of polynomial growth, but the doubling dimension is not controlled for notable important cases, such as for the permutation group S n (cf. section 3 of [17]) or for the group ( Z / 2 Z ) n . In order to cover these cases, it will be interesting to build analogues of our main result based on other group complexity bounds, beyond the doubling dimension case.

Acknowledgments and Disclosure of Funding

The authors thank Aryeh Kontorovich for readily answering questions (and supplying references) about Kolmogorov-Tikhomirov, and Bryn Elesedy for graciously answering queries and providing a copy of his dissertation before it appeared online. MP thanks the Centro Nacional de Inteligencia Artificial in Chile for support, as well as Fondecyt Regular grant number 1210462 entitled 'Rigidity, stability and uniformity for large point configurations'.

References

Yaser S. Abu-Mostafa. Hints and the VC Dimension. Neural Computation , 5(2):278-288, 03 1993.

Brandon Anderson, Truong Son Hy, and Risi Kondor. Cormorant: Covariant molecular neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Fabio Anselmi, Joel Z. Leibo, Lorenzo Rosasco, Jim Mutch, Andrea Tacchetti, and Tomaso A. Poggio. Unsupervised learning of invariant representations in hierarchical architectures. CoRR , abs/1311.4158, 2013.

Fabio Anselmi, Lorenzo Rosasco, and Tomaso Poggio. On invariance and selectivity in representation learning. Information and Inference: A Journal of the IMA , 5(2):134-158, 05 2016.

Minkyung Baek, Frank Dimaio, Ivan V. Anishchenko, Justas Dauparas, Sergey Ovchinnikov, Gyu Rie Lee, Jue Wang, Qian Cong, Lisa N. Kinch, R. Dustin Schaeffer, Claudia Millán, Hahnbeom Park, Carson Adams, Caleb R. Glassman, Andy M. DeGiovanni, Jose H. Pereira, Andria V. Rodrigues, Alberdina Aike van Dijk, Ana C Ebrecht, Diederik Johannes Opperman, Theo Sagmeister, Christoph Buhlheller, Tea Pavkov-Keller, Manoj K. Rathinaswamy, Udit Dalwadi, Calvin K. Yip, John E. Burke, K. Christopher Garcia, Nick V. Grishin, Paul D. Adams, Randy J. Read, and David Baker. Accurate prediction of protein structures and interactions using a three-track neural network. Science , 373:871 - 876, 2021.

Peter Bartlett. Covering numbers, chaining & dudley's entropy integral (lecture 24). class notes on statistical learning theory, 2006.

Arash Behboodi, Gabriele Cesa, and Taco S Cohen. A pac-bayesian generalization bound for equivariant networks. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 5654-5668. Curran Associates, Inc., 2022.

Erik J. Bekkers. B-spline cnns on lie groups. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020.

Alberto Bietti, Luca Venturi, and Joan Bruna. On the sample complexity of learning under geometric stability. Advances in neural information processing systems , 34:1867318684, 2021.

Olivier Bousquet, Stéphane Boucheron, and Gábor Lugosi. Introduction to statistical learning theory. Advanced Lectures on Machine Learning: ML Summer Schools 2003, Canberra, Australia, February 2-14, 2003, Tübingen, Germany, August 4-16, 2003, Revised Lectures , pages 169-207, 2004.

Anadi Chaman and Ivan Dokmanic. Truly shift-invariant convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3773-3783, 2021.

Shuxiao Chen, Edgar Dobriban, and Jane H Lee. A group-theoretic framework for data augmentation. The Journal of Machine Learning Research , 21(1):9885-9955, 2020.

Taco Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling. Gauge equivariant convolutional networks and the icosahedral cnn. In ICML , 2019.

Taco Cohen and Max Welling. Group equivariant convolutional networks. In ICML , 2016.

Taco S. Cohen. Learning transformation groups and their invariants. Master's thesis, University of Amsterdam, 2013.

Taco S Cohen, Mario Geiger, and Maurice Weiler. A general theory of equivariant CNNs on homogeneous spaces. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Michel Deza and Tayuan Huang. Metrics on permutations, a survey. Journal of Combinatorics, Information & System Sciences , 23(1-4):173-185, 1998.

Stephan Eismann, Raphael J. L. Townshend, Nathaniel Thomas, Milind Jagota, Bowen Jing, and Ron O. Dror. Hierarchical, rotation-equivariant neural networks to select structural models of protein complexes. Proteins: Structure , 89:493 - 501, 2020.

Bryn Elesedy. Provably strict generalisation benefit for invariance in kernel methods. Advances in Neural Information Processing Systems , 34:17273-17283, 2021.

Bryn Elesedy. Group symmetry in PAC learning. In ICLR 2022 Workshop on Geometrical and Topological Representation Learning , 2022.

Bryn Elesedy. Symmetry and Generalisation in Machine Learning . PhD thesis, University of Oxford, 2023.

Bryn Elesedy and Sheheryar Zaidi. Provably strict generalisation benefit for equivariant models. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 2959-2969. PMLR, 2021.

Marc Finzi, Gregory Benton, and Andrew G Wilson. Residual pathway priors for soft equivariance constraints. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 30037-30049. Curran Associates, Inc., 2021.

Marc Finzi, Max Welling, and Andrew Gordon Wilson. A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups. ArXiv , abs/2104.09459, 2021.

Jonathan Gordon, David Lopez-Paz, Marco Baroni, and Diane Bouchacourt. Permutation equivariant models for compositional generalization in language. In ICLR , 2020.

B. Haasdonk, A. Vossen, and H. Burkhardt. Invariance in kernel methods by haarintegration kernels. In Heikki Kalviainen, Jussi Parkkinen, and Arto Kaarna, editors, Image Analysis , pages 841-851, Berlin, Heidelberg, 2005. Springer Berlin Heidelberg.

Osman Semih Kayhan and Jan C van Gemert. On translation invariance in cnns: Convolutional layers can exploit absolute spatial location. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14274-14285, 2020.

Johannes Klicpera, Janek Groß, and Stephan Günnemann. Directional message passing for molecular graphs. ArXiv , abs/2003.03123, 2020.

Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In ICML , 2018.

Aryeh Kontorovich and Roi Weiss. Maximum margin multiclass nearest neighbors. In International conference on machine learning , pages 892-900. PMLR, 2014.

Robert Krauthgamer and James R Lee. Navigating nets: Simple algorithms for proximity search. In Proceedings of the fifteenth annual ACM-SIAM symposium on Discrete algorithms , pages 798-807. Citeseer, 2004.

Yann André LeCun, Bernhard E. Boser, John S. Denker, Donnie Henderson, Richard E. Howard, Wayne E. Hubbard, and Lawrence D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural Computation , 1:541-551, 1989.

Clare Lyle, Mark van der Wilk, Marta Z. Kwiatkowska, Yarin Gal, and Benjamin Bloem-Reddy. On the benefits of invariance in neural networks. ArXiv , abs/2005.00178, 2020.

Haggai Maron, Heli Ben-Hamu, Nadav Shamir, and Yaron Lipman. Invariant and equivariant graph networks. ArXiv , abs/1812.09902, 2019.

Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Learning with invariances in random features and kernel models. In Mikhail Belkin and Samory Kpotufe, editors, Proceedings of Thirty Fourth Conference on Learning Theory , volume 134 of Proceedings of Machine Learning Research , pages 3351-3418. PMLR, 15-19 Aug 2021.

Marvin Minsky and Seymour Papert. Perceptrons, Expanded Edition: An Introduction to Computational Geometry . The MIT Press, Cambridge, MA, 1987.

Behnam Neyshabur, Srinadh Bhojanapalli, David A. McAllester, and Nathan Srebro. A pac-bayesian approach to spectrally-normalized margin bounds for neural networks. ArXiv , abs/1707.09564, 2017.

Siamak Ravanbakhsh, Jeff G. Schneider, and Barnabás Póczos. Equivariance through parameter-sharing. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 , volume 70 of Proceedings of Machine Learning Research , pages 2892-2901. PMLR, 2017.

Marco Reisert and Hans Burkhardt. Learning equivariant functions with matrix valued kernels. Journal of Machine Learning Research , 8(15):385-408, 2007.

David W Romero and Suhas Lohit. Learning partial equivariances from data. Advances in Neural Information Processing Systems , 35:36466-36478, 2022.

Akiyoshi Sannai, Masaaki Imaizumi, and Makoto Kawano. Improved generalization bounds of group invariant / equivariant deep networks via quotient feature spaces. In Cassio P. de Campos, Marloes H. Maathuis, and Erik Quaeghebeur, editors, Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, UAI 2021, Virtual Event, 27-30 July 2021 , volume 161 of Proceedings of Machine Learning Research , pages 771-780. AUAI Press, 2021.

Victor Garcia Satorras, E. Hoogeboom, F. Fuchs, I. Posner, and M. Welling. E(n) equivariant normalizing flows for molecule generation in 3d. ArXiv , abs/2105.09016, 2021.

Bernhard Schölkopf, Chris Burges, and Vladimir Vapnik. Incorporating invariances in support vector learning machines. In Christoph von der Malsburg, Werner von Seelen, Jan C. Vorbrüggen, and Bernhard Sendhoff, editors, Artificial Neural Networks - ICANN 96 , pages 47-52, Berlin, Heidelberg, 1996. Springer Berlin Heidelberg.

H. Schulz-Mirbach. On the existence of complete invariant feature spaces in pattern recognition. In Proceedings., 11th IAPR International Conference on Pattern Recognition. Vol.II. Conference B: Pattern Recognition Methodology and Systems , pages 178-182, 1992.

Han Shao, Omar Montasser, and Avrim Blum. A theory of PAC learnability under transformation invariances. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

John. Shawe-Taylor. Building symmetries into feedforward networks. In 1989 First IEE International Conference on Artificial Neural Networks, (Conf. Publ. No. 313) , pages 158-162, 1989.

John Shawe-Taylor. Threshold network learning in the presence of equivalences. In NIPS , 1991.

John Shawe-Taylor. Symmetries and discriminability in feedforward network architectures. IEEE Transactions on Neural Networks , 4(5):816-826, 1993.

John Shawe-Taylor. Introducing invariance: a principled approach to weight sharing. In Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94) , volume 1, pages 345-349 vol.1, 1994.

John Shawe-Taylor. Sample sizes for threshold networks with equivalences. Information and Computation , 118(1):65-72, 1995.

Jure Sokolic, Raja Giryes, Guillermo Sapiro, and Miguel Rodrigues. Generalization error of invariant classifiers. In Artificial Intelligence and Statistics , pages 1094-1103. PMLR, 2017.

I. Sosnovik, A. Moskalev, and A. Smeulders. Scale equivariance improves siamese tracking. 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) , pages 2764-2773, 2021.

Behrooz Tahmasebi and Stefanie Jegelka. The exact sample complexity gain from invariances for kernel regression on manifolds, 2023.

Vladimir M. Tikhomirov. ε -entropy and ε -capacity of sets in functional spaces. Selected Works of AN Kolmogorov: Volume III: Information Theory and the Theory of Algorithms , pages 86-170, 1993.

Raphael J. L. Townshend, Stephan Eismann, Andrew M. Watkins, Ramya Rangan, Maria Karelina, Rhiju Das, and Ron O. Dror. Geometric deep learning of rna structure. Science , 373:1047 - 1051, 2021.

Tycho F. A. van der Ouderaa, David W. Romero, and Mark van der Wilk. Relaxing equivariance constraints with non-stationary continuous filters. In Advances in Neural Information Processing Systems (NeurIPS) , volume 35, Dec 2022.

Andrea Vedaldi, Matthew Blaschko, and Andrew Zisserman. Learning equivariant structured output svm regressors. In 2011 International Conference on Computer Vision , pages 959-966, 2011.

Ulrike von Luxburg and Olivier Bousquet. Distance-based classification with lipschitz functions. J. Mach. Learn. Res. , 5(Jun):669-695, 2004.

Rui Wang, Robin Walters, and Rose Yu. Approximately equivariant networks for imperfectly symmetric dynamics. In International Conference on Machine Learning , pages 23078-23091. PMLR, 2022.

Maurice Weiler, Patrick Forré, Erik P. Verlinde, and Max Welling. Coordinate independent convolutional networks - isometry and gauge equivariant convolutions on riemannian manifolds. ArXiv , abs/2106.06020, 2021.

Jennifer C. White and Ryan Cotterell. Equivariant transduction through invariant alignment. In Proceedings of the 29th International Conference on Computational Linguistics , pages 4651-4663, Gyeongju, Republic of Korea, Oct. 2022. International Committee on Computational Linguistics.

Marysia Winkels and Taco Cohen. Pulmonary nodule detection in ct scans with equivariant cnns. Medical image analysis , 55:15-26, 2019.

Jeffrey Wood and John Shawe-Taylor. Theory of symmetry network structure. Technical report, 1993.

Jeffrey Wood and John Shawe-Taylor. Representation theory and invariant neural networks. Discrete Applied Mathematics , 69(1):33-60, 1996.

Jeffrey Wood and John Shawe-Taylor. A unifying framework for invariant pattern recognition. Pattern Recognition Letters , 17(14):1415-1422, 1996.

Yinshuang Xu, Jiahui Lei, Edgar Dobriban, and Kostas Daniilidis. Unified fourier-based kernel and nonlinearity design for equivariant networks on homogeneous spaces. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 24596-24614. PMLR, 2022.

Sicheng Zhu, Bang An, and Furong Huang. Understanding the generalization benefit of model invariance from a data perspective. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 4328-4341. Curran Associates, Inc., 2021.

Xupeng Zhu, Dian Wang, Ondrej Biza, Guanang Su, Robin Walters, and Robert Platt. Sample efficient grasp learning using equivariant models. CoRR , abs/2202.09468, 2022.

A Proofs

A.1 Proofs of results from Section 4

A.1.1 Proof of Proposition 4.1

Proof. Reformulating (4.1) we find that with probability at least 1 -δ it holds that:

Next, we compound the bounds for R ( F Z ). The optimum α > 0 in the first line of (4.2) is the one for which n 9 = ln N ( F , α, ‖ · ‖ ∞ ). Keeping in mind (4.6), we choose α = diam ( Z ) n -1 ddim Z in (4.2), which gives (abbreviating d = ddim Z , D = diam Z )

Now the last equation and equation (A.1) yield the claim.

A.1.2 Proof of Proposition 4.2

We start with a preliminary result under hypotheses of strict equivariance. In this case, we are able to use a change of variables to reduce the generalization error formula to an equivalent one depending only on a measurable choice of G -orbit representatives of elements from Z :

Proposition A.1. Let F be a set of G -invariant functions, and let Z 0 ⊂ Z be a choice of G -orbit representatives for points in Z , such that ι 0 : Z → Z 0 associating to each z ∈ Z its orbit representative z 0 , is Borel measurable. Let F 0 := { f | Z 0 : f ∈ F} and denote by ι 0 ( D ) the image measure of D . Then for each n ∈ N if { Z i } n i =1 are i.i.d. samples with Z i ∼ D and Z 0 i := ι 0 ◦ Z i , we have

Proof. For the first equality, we use the definition of GenErr and the change of variable formula (3.1) and the fact that G -invariant functions f satisfy f ( Z ) = E g [ f ( g · Z )]. For the second equality, note that by hypothesis, for each f ∈ F we have f ( z ) = f ( g · z ) for all g ∈ G,z ∈ Z , in particular f ( z ) = f ( ι 0 ( z )) and we conclude by a change of variable by the map ι 0 in the expectations from the definition of GenErr ( G ) .

Now the proof Proposition 4.2 combines the above idea with a simple extra step:

Proof of Proposition 4.2: The proof uses the triangular inequality. For f ∈ F and g ∈ Stab /epsilon1 , we have:

By averaging over g ∈ Stab /epsilon1 , we obtain the inequality in the statement of the proposition. The equality follows by a change of variable via map ι 0 /epsilon1 , exactly as in the strategy of Proposition A.1.

A.1.3 Proof of Corollary 4.3 and of the more general result of Theorem A.3

In order to make the treatment better digestible, we first consider the intuitively simpler case of strict equivariance, and then describe how to extend it to the more general case of approximate and partial equivariance. In this case, if we restrict our equivariant functions to only the space of orbit representatives Z 0 , the dimension counts from classical generalization bounds of Proposition 4.1 improve as follows:

Corollary A.2. Assume that F is composed of G -invariant functions and that d 0 := ddim ( Z 0 ) > 2 . Also, denote D 0 := diam ( Z 0 ) . With the same notation as in Proposition A.1 and with the hypotheses of Proposition 4.1, for any probability distribution D over Z , the following holds with probability at least 1 -δ :

Proof. Due to Proposition A.1, we only need to bound GenErr ( F 0 , { Z 0 i } , ι 0 ( D )). Thus it suffices to apply Proposition 4.1 for the above function. We note that ‖F 0 ‖ ∞ ≤ ‖F‖ ∞ to conclude.

The drawback of the above Corollary, is that it leaves open the question of how to actually bound the diameter and dimension of Z 0 , on which we do not have direct control. The next steps we take consist precisely in translating the information from properties of G to relevant properties of Z 0 .

A first, simpler, approach could be the following. Under the reasonable assumption that Z , Z 0 have diameter greater than 1, the leading term on the left in Corollary A.2 is n -1 /d 0 .Thus the optimal choices of Z 0 are those which minimize the doubling dimension d 0 = ddim ( Z 0 ) amongst sets of representatives of G -orbits. This is a weak regularity assumption, implying that we want Z 0 to not oscillate wildly. The effect of G on coverings is evident in case G , Z 0 are manifolds, and Z = Z 0 × G (see (A.3) for the strictly equivariant case, and the more general (A.5) for the general case). Since ddim coincides with topological dimension, we immediately have

Intuitively, the dimensionality of G can be understood as eliminating degrees of freedom from Z , and it is this effect that improves generalization by n -1 / ( d -dim ( G )) -n -1 /d .

In order to include more general situations, we now describe a second, more in-depth approach. We take a step back and rather than addressing direct diameter and dimension bounds for Z 0 , we go "back to the source" of Proposition 4.1. We update the bounds on covering numbers of Z 0 , directly in terms of the G -action and of Z . The ensuing framework is robust enough to later include, after a few adjustments, also the cases of partial and approximate equivariance. Here is our fundamental bound, which generalizes and extends [51, Thm.3].

Theorem A.3. Assume that Z is a metric space with distance d and S ⊂ G is a subset of a metric group G consisting of transformations g : Z → Z (with action denoted g · z := g ( z ) ) for which there exists a choice of S -orbit representatives Z 0 ⊂ Z and a distance function d on S satisfying the following for L, L ′ ∈ (0 , + ∞ ] (with the conventions 1 / + ∞ := 0 and N ( X, 0) := + ∞ , N ( X, + ∞ ) := 0 ):

For all z 0 , z ′ 0 ∈ Z 0 and all g ∈ S it holds that 1 L d ( z 0 , z ′ 0 ) ≤ d ( g · z 0 , g · z ′ 0 ) ≤ L ′ d ( z 0 , z ′ 0 ) .

For all g, g ′ ∈ S and z 0 , z ′ 0 ∈ Z 0 it holds that 1 L d G ( g, g ′ ) ≤ d ( g · z 0 , g ′ · z ′ 0 ) ≤ L ′ d G ( g, g ′ ) .

Then for each δ ′ > 0 the following holds

Before the proof, we observe how Corollary 4.3 can be recovered using the choice L > 0 , L ′ = + ∞ in Theorem A.3:

Proof of Corollary 4.3: We apply Theorem A.3 to S = Stab /epsilon1 and Z 0 = Z 0 /epsilon1 as in the corollary statement. Then we take δ = 2 Lδ ′ in the conclusion (A.2), and consider only the lower bound inequality, which directly gives the claim of the corollary.

Proof of Theorem A.3: In this proof, we will denote a minimal α -ball cover of a metric space X by X α .

/negationslash

Note that we are not assuming G to be a group, but due to lower bound in property 1. it follows that z 0 ↦→ g · z 0 is injective (for z 0 = z ′ 0 we have d ( z 0 , z ′ 0 ) > 0 and thus g · z 0 = g · z ′ 0 ), and when below we write " g -1 " this has to be interpreted as the inverse of the g -action, restricted to its image.

/negationslash

Further, note that the case when one of L, L ′ is + ∞ , corresponds to removing the part of the assumptions (and of the conclusions) involving that value, thus we only consider the case of finite L ′ and finite L .

On fixing an arbitrary point z ∈ Z , we can write z = g · z 0 for a suitable g ∈ G,z 0 ∈ Z 0 . Let η := δ ′ / 2 L ′ . For fixed covers Z 0 η , G η , there exists a point z ′ 0 ∈ Z 0 η with d ( z ′ 0 , z 0 ) < η and g ′ ∈ G η with d G ( g ′ , g ) < η . Thus by property 1. we have d ( g · z ′ 0 , z ) < L ′ η = /epsilon1/ 2 and by property 2. we have d ( g ′ · z ′ 0 , g · z ′ 0 ) < L ′ η = δ ′ / 2. By the triangle inequality, d ( g ′ · z ′ 0 , z ) < δ ′ and thus G η · Z 0 η is an δ ′ -cover of Z . Thus we have

optimizing over the cardinalities on the right hand side yields the second inequality in (A.2).

Now consider an δ ′ -cover Z δ ′ of Z , and for η = 2 δL consider an η -cover G η of G . We find that for each z ∈ Z δ ′ , there exists at most one g ∈ G η such that dist ( z, g · Z 0 ) < η/ 2 L = δ ′ . Notice that if this were false, we could use the triangle inequality and contradict property 2. in the statement. For each g ∈ G η denote Z g the set of such points z ∈ Z δ ′ such that there exists x ∈ g · Z 0 , and assign exactly one such x = x ( z ) to each z , forming a set X g of all such x ( z ). Any other point x ′ ∈ g · Z 0 such that d ( x ′ , z ) < δ ′ then satisfies d ( x ′ , x ) < 2 δ ′ by triangle inequality, and thus X g is a 2 δ ′ -cover of g · Z 0 . If for g · z 0 ∈ g · X 0 the point x ∈ g · X 0 satisfies d ( g · z 0 , x ) < 2 δ ′ , then by property 1. in the statement we have d ( z 0 , g -1 · x ) < 2 δ ′ L , and thus g -1 · X g is a 2 δ ′ L -cover of Z 0 , having the same cardinality as Z g . We then compute as follows, proving the first inequality in (A.2):

A.1.4 Proof of Theorem 4.4

As before, we focus again first on the exact equivariance case, where Theorem A.4 is the direct analogue to (or special case of) Theorem 4.4.

Under the hypotheses of Theorem A.3 on the G -action, we directly obtain the following, for the strictly equivariant case:

We next impose that for δ ≲ diam ( G ) the group G satisfies the natural "volume growth" assumption, where for compact groups Vol ( G ) is its dim ( G )-dimensional Hausdorff measure and dim ( G ) is the usual Hausdorff dimension, and for finite groups we use minimum separation notation δ G > 0 as defined in (4.4):

Similarly to Proposition 4.1, we then get the following, in which the leading term in the bound has exponent figuring d 0 = ddim ( Z ) -dim ( G ). Recall that dim ( G ) = 0 for finite groups, thus the distinction can be made directly in terms of the dimension of G .

Theorem A.4. Let δ > 0 be fixed. Assume that for a given ambient group G the group G satisfies assumption (A.4) and that its action satisfies the the assumptions 1. and 2. of Theorem A.3 for some finite L > 0 and L ′ = + ∞ . We denote d G = ddim ( G ) if G is a discrete group and d G = dim ( G ) if G is compact and non-discrete, and d = ddim ( Z ) . Furthher assume d 0 := d -d G > 2 . Furthermore, set | G | := Vol ( G ) if dim G > 0 and | G | := # G for finite G . Then with the notations of Proposition 4.1 we have with probability ≥ 1 -δ

where

Proof. We follow the same computation as Proposition 4.1, but use Proposition A.1 in order to reduce to restrictions of functions to Z 0 . In this case, using (A.3) and assumption (A.4), and with notation as in our statement, we will have:

where we have δ G = 0 for dim G > 0. We set C := (2 L ) d D d | G | for simplicity of notation. In case Cδ d 0 G < 1 (which includes the case dim G > 0), we take α = ( C/n ) 1 /d 0 in the Dudley integral (4.2) and find

from which the proof goes exactly as in Proposition 4.1, with C replacing D d , and we get the second option for the value of ( E ) as given in our statement. In case Cδ d 0 G < 1 instead we take α = 0 and our above bound for N ( Z 0 , t ) plugged into (4.2) (recalling the notation for C ):

from which the second case of the value of ( E ) follows by direct computation.

Now the proof of Theorem 4.4 proceeds in exactly the same manner as the above. Below we explain the required adaptations:

Proof of Theorem 4.4: The following updates are the principal adaptations required for the above proof of Theorem A.4:

The role of G should be replaced by Stab /epsilon1 , except for the fact that parameters δ G , d G remain unchanged (i.e. we use their values corresponding to "ambient" group G rather than those for Stab /epsilon1 ).

The G -orbit representative set Z 0 then should be replaced by representatives Z 0 /epsilon1 for orbits of Stab /epsilon1 .

With these changes, assumption (A.4) implies its more general version, assumption (4.7). Indeed, | G | equals # G for finite G and Vol ( G ) for compact G , and δ G > 0 only in the first case. Furthermore, we have δ Stab /epsilon1 ≥ δ G as a direct consequence of Stab /epsilon1 ⊆ G .

We observe that Corollary 4.3 (which also is obtained from Theorem A.3 with the above two main substitutions) directly gives the version of Theorem (A.3) required to get the correct replacement of (A.3) under our initially declared two substitutions. We get:

With the above changes, the proof follows by exactly the same steps as in the above proof of Theorem A.4.

Remark A.5 . Note that, as might be evident from the last proof, we could have introduced new more precise parameters to keep track of dimensionality and minimum separation for Stab /epsilon1 rather than formulating assumption (4.7) in terms of d G , δ G . This is justified for the aims of this work. Indeed, all the main situations of interest to us are those in which Stab /epsilon1 is a "large" subset of G , i.e. it has dimension d G , and in all our examples for finite groups δ G > 0, the minimum separation for Stab /epsilon1 is within a small factor of δ G itself.

A.2 Proof of Proposition 5.1

Proof. We have that Z = ( X,Y ) is a data distribution in which by our assumption 2. preceding the proposition, we have that almost surely Y = y ∗ ( X ) for a deterministic function y ∗ . With this notation, we may write

Recalling that we restrict to functions of the form f ( x, y ) = /lscript ( ˜ f ( x ) , y ) = d Y ( ˜ f ( x ) , y ) 2 , we first consider the precise equivariance case /epsilon1 = 0. In this case for g ∈ Stab /epsilon1 ( F ) we find also ˜ f ( g · X 0 /epsilon1 ) = g · ˜ f ( X 0 /epsilon1 ) and thus when optimizing over ˜ f we have to determine the optimal value of y = ˜ f ( X 0 /epsilon1 ) to be associate to each X 0 /epsilon1 . Thus as a consequence of all the above, if ˜ F would be the class of all precisely Stab /epsilon1 -equivariant measurable functions, we would get the following rewriting:

For /epsilon1 > 0, for each fixed X 0 /epsilon1 = x 0 /epsilon1 we may further perturb the associated y = ˜ f ( x 0 /epsilon1 ) by at most /epsilon1 in the direction of y ∗ ( X 0 /epsilon1 ), while still obtaining a measurable function with approximation /lscript ∞ -norm error bounded by /epsilon1 , thus the above bound is improved to

as desired. In case ˜ F contains a strict subset of measurable invariant functions with error /epsilon1 , we would only get an inequality instead of the first equality in (A.6) but we still have the same bound as in (A.7), and thus the proof is complete.

A.3 Proof of Theorem 5.2

Proof. We use the isodiametric inequality (5.1) in G , applying it to Stab /epsilon1 ( F ) for F ∈ C /epsilon1,λ . Then by taking Stab /epsilon1 ( F ) = X which is optimal for inequality (5.1) we can saturate the two bounds (modulo discretization errors for discrete G ) and we get

We next use Lipschitz deformation bounds and find that for all x ∈ X we have

Then we use Proposition 5.1 for F and observe that when g, X 0 /epsilon1 are random variables as in the proposition, in particular g ∈ Stab /epsilon1 ( F ) and for each X 0 /epsilon1 = x 0 /epsilon1 we find the following estimate valid uniformly over y ∈ { y ∗ ( g · X 0 /epsilon1 ) : g ∈ Stab /epsilon1 ( F ) } :

In a similar way, we also find

By triangle inequality, and using the assumption that Lip ( y ∗ ) /similarequal 1 it follows that

Then we may perturb each y by /epsilon1 in order to possibly diminish this value without violating the condition defining Stab /epsilon1 ( F ), and with these choices we obtain the claim.

A.4 Finding the optimal λ = λ ∗ for the bound of Theorem 6.1

We note that λ ∗ minimizing Cλ α + C ′ λ -β for α, β > 0 is given by

recall that in our case have the following choices, for case 1 and case 2 in the theorem's statement.

and thus the optimal choice of λ is

B Examples

We describe some concrete examples of partial and approximate equivariance using the language we used in section 3.2 while sourcing them from existing literature. But first, we expand a little on our equivariance error notation.

B.1 Equivariance error notation

Recall that the action of elements of an ambient group G over the product space Z = X ×Y may be written as follows: For coordinates z = x × y we may write g · z = ( g · x, g · y ), and thus for ˜ f : X → Y and f ( x, y ) = /lscript ( ˜ f ( x ) , y ), we have the action

For the equivariance error of g, f , interpreted as "the error of f 's approximate equivariance under the action by g ", we get the following, which is valid in the common situations in which /lscript ( y, y ′ ) ≥ 0 in general with /lscript ( y, y ) = 0 for all y ∈ Y :

where the last inequality follows by restricting the supremum from X × Y to the graph of ˜ f , namely by imposing y = ˜ f ( x ).

In several recent works, the equivariance error is defined simply by comparing g · ˜ f ( x ) and ˜ f ( g · x ), as in the rightmost term of (B.1), thus it is lower than the one found here. We provide a justification for our definition of the equivariance error:

The loss /lscript is the integrative part of the model, thus a definition for equivariance error which does not include it will only detect partial information concerning the influence of symmetries.

The notion of equivariance error defined via /lscript simplifies the comparison between ˜ f and data distributions D .

and

B.2 Examples

B.2.1 Imperfect translation equivariance in classical CNNs

We consider here the most common examples of group equivariant convolutional networks (GCNNs), which are the usual Convolutional Neural Networks (CNNs) for computer vision tasks. We follow observations from [27] and [11], and connect the underlying ideas to Theorem 6.1.

Setting of the problem. We consider a usual CNN layer, keeping in mind a segmentation task, where both X = Y represent spaces of images. More precisely, we think of images as pixel values at integer-coordinate positions taken from the index set Z × Z . We also assume that the relevant information of each image only comes from a square of size n × n pixels, outside which the pixel values are taken to be 0. We consider the application of a single convolution kernel/filter, of k × k pixels (with k a small odd number). One typically applies padding by a layer of 0's of size ( k -1) / 2 on the perimeter of the n × n square, after which convolution with the kernel is computed on the n × n central pixels of the resulting ( n + k -1) × ( n + k -1) padded input image. The output relevant information is restricted to a n × n square, outside which pixel values are set to 0 again, via padding.

Metric on X . As a natural choice of distance over X we may consider L 2 -difference between pixel-value functions, or interpret pixel values as probability densities, and use Wasserstein distance, or consider other ad-hoc image metrics.

Group action: translations. The group acting on our 'pixel-value functions' is the group of translations with elements from Z × Z . We expect the following invariance for the segmentation function f : X → X :

where x ∈ X represents an image with pixel values assigned to integer coordinates and v ∈ Z 2 is a translation vector and (in two alternative notations) v · x = τ v ( x ) is the result of translating all pixel values of x by v .

Deformation properties of the action. If we take the previously mentioned distance functions on X and the usual distance induced from R 2 over translation vectors v , it is easy to verify that the assumptions of Theorem A.3 about the action of translations hold, and the Lipschitz constants with respect to the metric on X only depend on the mismatch near the boundary, due to 'zero pixels moving in' and to 'interior pixels moving out' of our n × n square, and being truncated. The ensuing bounds only depend on the precise distance that we introduce use on X .

More realistic actions. An alternative more realistic definition of Z × Z -action consists of defining v · x as the truncation of τ v ( x ) where, for pixels outside our 'relevant' n × n square we set pixel value to 0 after the translation.

Problems near the boundary. Nevertheless, the updated translation action, will move pixel values of 0, coming from pixels outside the n × n square, and will create artificial zero pixel values inside the image v · x , different than the values that would be present in real data.

Imperfect equivariance of data. Also, even in the latter more realistic alternative, the above translation equivariance is not respecting by segmentation input-output pairs coming from finite n × n images, since, independently of n , the boundary pixel positions translated by v , fall outside the original image.

Approximate stabilizer. In any case, we have to restrict the choices of v to integercoordinate elements of a (subset of a) n × n square, containing only the translations that are relating 'real' segmentations that appear within our n × n relevant window. It is thus

natural to restrict Stab /epsilon1 to only include vectors in a smaller subset of Z × Z , of cardinality

The value of error /epsilon1 may quantify the allowed error (or noisiness) for our model sets.

Reducing to finite G and computing λ . For the sake of computing λ in our Theorem 6.1, we can observe that input and output values outside the 'padding perimeter' given by a ( n + k -1) × ( n + k -1) square, are irrelevant, and thus we may actually periodize the images and consider the images as subsets of a padded torus ( Z / ( n + k -1) Z ) 2 , which we consider as acting on itself by translations. In this case G /similarequal ( Z / ( n + k -1) Z ) 2 , so that

Further extensions. In [27], it is argued that convolutional layers in classical CNNs are not fully translation-equivariant, and can encode positional information, due to the manner in which boundary padding is implemented. Possible solutions are increasing the padding to size k (so that the padded image is a square of size ( n +2 k ) × ( n +2 k )) or extending images by periodicity, via so-called "circular padding" (which transforms each image into a space equivalent to the torus ( Z /n Z ) 2 ). In either case, the application of actions by translations by vectors that are too large compared to the image size of n × n , will increase the mismatch between model equivariance and data equivariance.

Stride and downsampling. In [11], a different equivariance error for classical CNNs is studied, related to the use of stride > 1 in order to lower the output dimensions of CNN layer outputs. If for example we use stride 2 when defining layer operation f : X → Y , then Y will have relevant pixel values only in an n/ 2 × n/ 2-square, and we apply the k × k convolution kernel only at positions with coordinates in (2 Z ) × (2 Z ) from image x . In this case we require that translations by group elements v ∈ (2 Z ) × (2 Z ) on x have the effect of a translation by v/ 2 ∈ Z on the output. However for shifts in the input via vectors v that do not have two even coordinates, we may not have have an explicit corresponding action on the output, and in [11] a solution via adaptive polyphase sampling is proposed. A possibility for studying the best polyphase sampling strategy via almost equivariance, would be to include a bound for equivariance error /epsilon1 > 0 and consider the optimization problem of finding the polyphase approximation that minimizes theoretical or empirical quantifications of /epsilon1 . As a benchmark (modelled on the case of infinite images without boundary effects) one could compare the above to the action via Stab 0 = (2 Z ) × (2 Z ) which has λ = 1 / 4 within the ambient group G = Z × Z . Our Theorem 6.1 can be used to compare the effects of increasing or decreasing /epsilon1,λ , in terms of data symmetry.

B.2.2 Partial equivariance in GCNNs

In this section, we connect the main results from [40] to our setup. In [40], one of the main motivating examples was to consider rotations applied to a handwritten digit and revert them. The underlying group action was via SO (2) and only rotations of angles between [ -60 ◦ , 60 ◦ ] were permitted in one case, which allowed to not confound rotated digits '3' and '9' for example.

The above task can be formulated on a space of functions f : X → Y in which X represents the space of possible images and Y the labels. Elements ( Y d , Y θ ) ∈ Y include a digit classification label Y d and a rotation angle value Y θ .

We consider actions by group G = SO (2) = { R φ : φ ∈ R / 360 Z } , where R φ is the rotation matrix by angle φ , and the group operation corresponds to summing the angles, modulo 360 ◦ (or in radians, modulo 2 π ). The action of R φ over X would be by rotation as usual ( R θ sends image x ∈ X to R θ · x , now rotated by θ ), and over Y we consider the action by

i.e. the restriction of the action on the digit label leaves it invariant and the restriction of the action on the angle label is non-trivial, giving a shift on the label.

The optimum labelling assigns to x a label y ∗ ( x ) enjoying precise equivariance under the above definitions of the actions, and thus we are allowed to permit equivariance error /epsilon1 = 0. However as mentioned above, applying rotations outside the range θ ∈ [ -60 ◦ , 60 ◦ ] to the data would surely bring us outside the labelled data distribution, thus we are led to take

We then have that | Stab 0 | / | SO (2) | = 1 / 3, with respect to the natural Haar measure on rotations. It is natural to think of the set of Stab 0 -action representatives of images X 0 0 given as the "unrotated" images. If we take a digit image that is rotated, say by 20 ◦ , from its 'base' version, and we apply a rotation of 50 ◦ to it (i.e. an element of Stab 0 ), then we reach the version of the image now excessively rotated by 70 ◦ . This means that without further modification, considering model symmetries with Stab 0 taken to be independent of the point, would automatically generate some error when tested on the data. While decreasing the threshold angle in the definition of Stab 0 from 60 ◦ would limit this effect, it will also decrease generalization error in the model. The study of point-dependent invariance sets Stab /epsilon1 is interesting in view of this example application, but it is outside the scope of the current approximation/generalization bounds and is left for future work.

B.2.3 Possible applications to partial equivariance in Reinforcement Learning

The use of approximate invariances for RL applications was considered in [23, Sec. 6] via soft equivariance constraints allowing better generalization for an agent moving in a geometric environment. While imposing approximate equivariance for memoryless G -action for groups such as G = SO (2) , Z 2 has produced positive results, it may be interesting, in analogy to the previous section, to include memory, and thus restrict the choices of group actions across time steps. Note that for a temporal evolution of T steps, the group action by G acting independently at each step would produce a T -interval action via the product group G T , and allowing for a partial action via Stab /epsilon1 , with possibly increased fitness to evolving data. More precise time-dependence prescriptions and consequences within Q -learnig are left to future work.

C Discussion of [20]

In section 2 we mentioned [20], which considers PAC-style bounds under model symmetry. [20] works with compact groups and argues how the learning problem in such a setup reduces to only working with a set of reduced orbit representatives, which leads to a generalization gain. This message of [20] is similar to ours, although we work with a more general setup. However, we noted that the main theorem in [20] has an error. Here, we briefly sketch the issue with the proof.

One of the main quantities of interest in the main theorem of [20] is D τ ( X , H ) (notation from their paper), which directly comes from the following bound, and is claimed to have a linear dependence on Cov ( X , ρ, δ ). Again, for the sake of easier verification, we follow their notation. Crucially, note that [20] uses the notation Cov as analogous to our N :

However, the correct application of the Kolmogorov-Tikhomirov estimate shows that the reasoning in the proof should yield a dependence which is exponential in Cov ( X , ρ, δ ), not linear. To see this, set s = 2 (sufficient for our purposes) in equation 238 in [54] (page 186). In other words, it is not possible to cover Lipschitz functions in infinity norm by only using constant functions.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a theoretical investigation into the approximation and generalization errors for a model exhibiting approximate equivariance. The paper introduces the concept of \"error to equivariance,\" a measure of the deviation of the error function from perfect equivariance. It then defines the $\\epsilon$-stabilizer, a subset of a group under whose transformations the error function maintains approximate equivariance. Leveraging these definitions, the paper dives into the analysis of generalization error for an approximately equivariant model in terms of PAC-bounds and Rademacher complexity, after which, It also explores the approximation error. Ultimately, the performance error of a model is defined as the sum of its generalization error and approximation error, providing a quantitative measure of the performance of an approximate equivariant model. In the end, the performance error can quantitatively analyze the trade-off between its two terms (generalization error and approximation error).\n\nSTRENGTHS:\n1. The paper theoretically studies approximate equivariance, which is a very significant problem in the equivariant learning literature as real-world problems are not always perfectly equivariant. \n2. The paper provides general bounds, contributing valuable theoretical tools to the field.\n\nWEAKNESSES:\n1. Section 4 is somewhat challenging to read, especially for readers who are not familiar with PAC-bounds and Rademacher complexity. Additional explanations or intuitive examples could improve accessibility.\n2. While not a requirement for a theory paper, the absence of empirical validation or experiments leaves a gap. Experiments that align with the theoretical bounds could provide practical guidance on how the paper's findings might be applied empirically.\n\nQUESTIONS:\n1. Could the authors comment on the broader implications of the proposed theory? Specifically, could the theory help model selection under approximate equivariance? \n2. The paper is titled \"Approximation-Generalization Trade-offs\", however, the high-level message regarding this trade-off isn't clear. \n3. In line 140, there is a closing parenthesis but no opening parenthesis.\n\nLIMITATIONS:\nThe paper properly discusses its limitations. An additional limitation to consider is the assumptions of the paper. The paper makes a couple of assumptions throughout the theoretical analysis, it might be good to summarize and discuss those assumptions in the limitation section.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-02"}

comments: SUMMARY:
In this paper, the authors spell out explicitly the benefit of capturing task specific symmetries into machine learning models. This goal is achieved by deriving rigorously  generalization error bounds that takes into account the cardinality of the set of transformations that a model is (approxaimately) equivariant to (Stab_\epsilon) as well as the group dimensionality these transformations are elements in.
The results are very general and unlike previously presented results in the literature, the sets of transformation the model is equivariant to, are not required to form a group.

STRENGTHS:
The presented approach to derive the bounds is principled and very interesting.

WEAKNESSES:
While the derived results are very general and rigorous, it is not directly obvious how these results can be applied. Although the authors give examples in the appendix, the impact of these bounds seems relatively limited. It could have been maybe useful to show an actual model performance w.r.t. the presented bound.

QUESTIONS:
Is there anything that can be said about the tightness of the presented bounds?

LIMITATIONS:
The authors state that the derived bounds are not model specific and stronger versions could be the subject of future work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides theoretical investigation on the intuition that, machine learning models (e.g. neural networks) whose symmetry aligns with the data symmetry achieve best performance and generalization. The paper formulate the relationship between model equivariance error and data equivariance error by providing quantitative bounds, which are applicable in the presence of approximate or partial data/model equivariance.\n\nSTRENGTHS:\n1. To my best knowledge, the paper provides a systematic and comprehensive review of the historical development of neural networks with group symmetries (**since this is not exactly my area of research, please correct me if I'm wrong**).\n\n2. The paper provides detailed and systematic analyses of the model generalization bounds as well as approximation bounds under the more general case of partial or approximate equivariance. According to the authors, as well as to my best knowledge, this is the first theoretical work that addresses this problem. Therefore its contribution and novelty is significant.\n\n3. Although not being able to check every technical details and derivations, I find the paper well-structured and easy to follow.\n\nWEAKNESSES:\n**Major issues:**\n\n1. Derivation of the approximation error bounds:\n\n- First of all, in line 291, I don't think it's legal to write $\\min A - \\min B$ as $\\min(A-B)$, since the second term has a minus sign. So I think the correct way to write is that\n\n    $\\text{AppGap}(\\mathcal{F}, \\mathcal{D}) := \\underset{f\\in\\mathcal{F}}{\\min}\\mathbb{E}[\\mathbb{E}_g[f(g\\cdot Z)]] - \\underset{F\\in\\mathcal{M'}}{\\min}\\mathbb{E}[F(Z)] \\ge \\underset{F\\in\\mathcal{M'}}{\\min}\\mathbb{E}[\\mathbb{E}_g[F(g\\cdot Z)]] - \\underset{F\\in\\mathcal{M'}}{\\min}\\mathbb{E}[F(Z)] $\n\n- The expression above provides a **lower bound** to the approximation gap, so how does the inequality become equality in Line 304? If it is derived by assuming $\\mathcal{F} = (\\mathcal{M}^{(G)})'$ (assumption 3 in line 301=303), then line 304 should be written as\n\n    $\\text{AppGap}((\\mathcal{M}^{(G)})',\\mathcal{D}) = \\underset{F\\in(\\mathcal{M}^{(G)})'}{\\min}\\mathbb{E}_Z[\\mathbb{E}_g[F(g\\cdot Z)]]$\n\n- Moreover, if the upperbound in Prop 5.1. is derived based on the equation above, I think it's necessary to prove that $(\\mathcal{M}^{(G)})'\\subseteq \\mathcal{C}_{\\epsilon,\\lambda}$ , for any $\\epsilon\\ge 0, \\lambda\\in(0, 1]$, which I don't see in the proof.\n\n2. Interpretation of the \"generalization-approximation trade-off\"/\"optimal equivariance\" is not clear:\n\n    In line 360-362, it is claimed that the optimal $\\lambda^*$ \"validates the intuition stated at the onset that for the best model its symmetries must align with that of the data\". However, based on my understanding, the symmetries of the model **only fully align with those of the data** when $\\lambda^*=1$, given $\\epsilon=0$. When $0<\\lambda^*<1$, I think it means the model only **partially preserves the symmetries of the data**. Do you agree? If this is the case, how do you justify the intuition that \"the integration of relevant symmetry results in enhanced generalization.\"?\n\n3. Similar to reference 9 in the paper, I think one key point I would expect to see in this paper is proving that, models with aligned group symmetries of data exhibit higher sample efficiency/complexity, which can generalize to scenarios of partial/approximate symmetry. I see no discussion or proofs regarding this intuition in the paper.\n\n  **I will be happy to raise my score given satisfactory explanation or revision.**\n\n**Minor Issues:**\n\n1. This paper targets specifically on the theory of machine learning models with partial/approximate group symmetries, which might be of interests only to a very specific subgroup of the whole machine learning community.\n\n2. There are some presentation issues need to be improved:\n\n    - All hyperlinks do not work when clicked. Some numberings are also confusing, e.g. what is Thm A.4 in line 242, 246, 268?\n\n    - Some of the statements would benefit from more citations/examples, e.g.\n     \"It has often been noted that this may impose an overly stringent constraint as real-world data seldom exhibits perfect symmetry” (line 36 - 37)\n3. Typos\n\n    - may have not -> may not have (line 312)\n\n    - $\\text{Stab}{\\epsilon}(\\mathcal{F'}) \\Longrightarrow \\text{Stab}{\\epsilon}(\\mathcal{F})$ (line 309)\n\nQUESTIONS:\n1. More regarding the interpretation of the \"generalization-approximation trade-off\"/\"optimal equivariance\":\n\n    If we examine the contribution of $\\lambda$ to the generalization and approximation bounds given by Theorem 4.4 and 5.2 respectively, assuming perfect symmetry of data ($\\epsilon=0$), we see that the former decreases when $\\lambda$ increases ($\\frac{1}{\\lambda^{1/2}}$ or $\\frac{1}{\\lambda^{1/d_0}}$ scaling), while the latter increases when when $\\lambda$ increases ($\\lambda^{1/d_G}$ scaling), which gives what the authors called \"generalization-approximation trade-off\". To me, this seems to show that minimizing the generalization error effectively aligns the model and data symmetries, whereas minimizing the approximation error functions in the opposite direction, resulting in a partially aligned model symmetry. \n\n    Do you agree with this analysis? Any further explanation/intuition?\n\n2. Can you explain what you mean by \"oscillations\", which is mentioned repeatedly? Example contexts like:\n\n- \"since $\\text{Stab}_{\\epsilon}$ includes more elements of G as ε increases, and we can expect that actions of elements generating higher equivariance error in general have higher oscillations\". (line 274-275)\n\n- line 320-322\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn their theoretic work, the  authors presents quantitative bounds on the generalization error and approximation error. They extend the results to approximate symmetries, which refers to functions that are close, in the supremum norm, to a group action.\n\nTheir first results shows that when working with only the representatives of $G$-Orbits, and the models are equivariant (or almost-equivariant) The generalization error can be nearly-recovered.  Their second results show that their exists a class almost-equivaraint estimators, such that its approximation error can be bounded (and the bound depends mostly on the properties of G, and somewhat on the pertubations of the optimal estimator).\n\nSTRENGTHS:\nThis paper is a novel and well presented work on approximate symmetry. The bounds presented are a significant contribution to the field, further deepening its theoretical foundations.\n\nWEAKNESSES:\n- The discussion section is lacking. It focuses on combining previous results and quantifying the lambda parameters. It is lacking interpretation of the results, and addressing your hypothesis. Under which conditions equivariance error is optimal? What is the interplay between data equivaraince to model equivariance? How does it compare to other bounds in the literature? \n- The limitation of this paper is not addressed properly. The authors only mentions that \"the results are not model specific\", which is not an actual limitation. It would be beneficial to discuss the actual limitation of the work. Can the bounds be computationally intractable? for example, how can one compute $ddim(G)$? What is the asymptotic growth of the presented bounds? \n    \n- The paper is lacking intuition into the $\\lambda$ parameter. Explanations are postponed into the discussion, and  even there, the interplay between $\\lambda$      and the error is drawn, but there is no attempt to demystify this parameter. I would appreciate further elaboration about it, and about the meaning of $Dens(\\epsilon)$. Is this the same as 'approximate orbits'$^{-1}$?\n\nQUESTIONS:\n- Theorem 5.2 holds under the condition that there \\textbf{exists} $\\mathcal{F} \\in \\mathcal{C}_{\\varepsilon, \\lambda}$ such that the theorem holds. I am concerned that ideally, we would like it to apply to \\textbf{any} such $\\mathcal{F}$ . It would seem to me that it would align better with our original motivation as we would like to prove that any such partial symmetry family (with limited perturbation) can generalize well. I note that theorem 6.1 is under the same assumption so my question additionally applies to it\n-  I would appreciate further elaboration specifically on the interplay between model equavarince against data equavarince. How does the presented bound connect between them? can you show how further restriction of the models, choosing $\\mathcal{F}' \\subset \\mathcal{F}$ would affect the generalization and approximation bound.\n\nLIMITATIONS:\n.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe submission discusses the role of task-specific symmetries in enhancing machine learning models. It presents a quantitative analysis substantiating the common intuition that task-specific symmetries lead to improved generalization. The authors delve into the interplay between model and data symmetry, particularly in the case of partial or approximate symmetries, establishing a defined link between model equivariance error and data equivariance error. They outline the conditions that lead to the optimal model equivariance error. However, the results are general and not model-specific, with the authors suggesting future research to formulate more specific versions of their theorem tailored to specific groups and neural network architectures.\n\nSTRENGTHS:\nThe paper provides a rigorous, quantitative exploration of task-specific symmetries in machine learning models.\n\nWEAKNESSES:\nThe presentation of the paper's findings could be clearer, particularly with regard to illustrating their novelty and applicability, and the frequent use of different notations without a consolidated reference adds unnecessary complexity and potential confusion for readers.\n\nQUESTIONS:\nThe paper's main contributions could be made more explicit. As it stands, the authors appear to enumerate their findings without adequately illustrating their novelty, importance, or applicability. Furthermore, the usage of numerous different notations creates some confusion; a table of notation might ameliorate this issue. While the paper seems to address a fundamental problem in machine learning theory, its presentation could be improved, and the clarity of its contributions could be enhanced.\n\nQuestions:\n\n1. The majority of results in the paper are upper bounds, but there is no discussion regarding their tightness. Could the authors elaborate on this aspect?\n\n2. In Lines 351-352, the authors mention that \"the above bounds are not informative.\" If the bounds lack informativeness, could the authors clarify the significance or utility of these results?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThere are two main results in this paper. The first one (Theorem 4.4) obtains a PAC-generalization bound for a hypothesis space which is approximately equivariant to transformations by elements g in G. This bound shows that in addition to the classical complexity term that depends upon the size of the hypothesis space, there are two additional terms that arise, one due to the approximate invariance (where one restricts to only the hypotheses that are equivariant to G) and another due to partial equivariance (where one relaxes the error to equivariance constraint). The proof of this result is classical (e.g., it is close to the Baxter’s proofs in “A Model of Inductive Bias Learning“).\n\nThe second result (Theorem 5.2) is about studying the interplay between equivariance imposed in the model and that possessed in the data. The key idea here is to construct a set of hypothesis spaces that capture a large subset of the \"symmetries\" in the data G, depending upon a parameter lambda; larger the lambda, smaller the set of hypotheses. This allows an upper bound on the approximation error in terms of lambda and the dimension of G.\n\nSTRENGTHS:\nThis is a thorough and pertinent paper that asks the question: how does the equivariance baked into the model relate to the inherent symmetries present in the data. This is a rare perspective. The proofs are somewhat straightforward but they are well-formulated and rigorous.\n\nWEAKNESSES:\nThe writing in Section 5 can be improved a lot.\n\nI thought that the discussion in Theorem 6.1 could be done in a different way. Specific values of lambda and epsilon entail a choice of hypothesis space. The constant M in Theorem 6.1 therefore is smaller if lambda is larger (?).\n\nMinor\nI suspect the \\in S on line 251 is a typo.\ntilde cal F on line 289 and 298 should be just cal F I think.\ncal F’ on line 309 should be cal F.\nThere is clearly a typo on Line 310.\nC_G should be defined before Line 327.\nTheorem 6.1 should have n^{-1/2} for the first term.\n\nQUESTIONS:\nIt would be useful to demonstrate a meaningful experiment (even if it is with synthetic data) where the three terms in Theorem 6.1 trade off against each other.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-31"}