SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\operatorname {S t a b} _ {\epsilon} = \operatorname {S t a b} _ {\epsilon} (\mathcal {F}) := \bigcap_ {f \in \mathcal {F}} \operatorname {S t a b} _ {\epsilon} (f).
$$
We note that the strict equivariance case is recovered if, for  $\epsilon = 0$ , we have  $\mathsf{Stab}_0 = G$ .
Proposition 4.2. Let  $\mathsf{Stab}_{\epsilon}$  be as above, and assume that  $|\mathsf{Stab}_{\epsilon}| > 0$ . let  $\mathcal{Z}_{\epsilon}^{0} \subset \mathcal{Z}$  be a measurable choice of  $\mathsf{Stab}_{\epsilon}$ -orbit representatives for points in  $\mathcal{Z}$ , and let  $\iota_{\epsilon}^{0}: \mathcal{Z} \to \mathcal{Z}_{\epsilon}^{0}$  be the measurable map that associates to each  $z \in \mathcal{Z}$  its representative in  $\mathcal{Z}_{\epsilon}^{0}$ . Let  $\mathcal{F}_{\epsilon}^{0} := \{f|_{\mathcal{Z}_{\epsilon}^{0}}: f \in \mathcal{F}\}$  and let  $\iota_{\epsilon}^{0}(\mathcal{D})$  represent the image measure of  $\mathcal{D}$ . Then for each  $n \in \mathbb{N}$ , if  $\{Z_i\}_{i=1}^n$  are i.i.d. samples with  $Z_i \sim \mathcal{D}$  and  $Z_{i,\epsilon}^{0} := \iota_{\epsilon}^{0} \circ Z_i$ , then we have
$$
\operatorname {G e n E r r} \left(\mathcal {F}, \left\{Z _ {i} \right\}, \mathcal {D}\right) \leq 2 \epsilon + \operatorname {G e n E r r} ^ {\left(\operatorname {S t a b} _ {\epsilon}\right)} \left(\mathcal {F}, \left\{Z _ {i} \right\}, \mathcal {D}\right) = 2 \epsilon + \operatorname {G e n E r r} \left(\mathcal {F} _ {\epsilon} ^ {0}, \left\{Z _ {i, \epsilon} ^ {0} \right\}, \iota_ {\epsilon} ^ {0} (\mathcal {D})\right).
$$
The above result says that the generalization error of our setting of interest could roughly be obtained by working with a reduced set of only the orbit representatives for points in  $\mathcal{Z}$ . This is in line with prior work such as [51, 20, 41]. However, note that our result is already more general and does not require that the set of transformations form a group. With this, we now need an understanding of the covering of spaces of representatives  $\mathcal{Z}_{\epsilon}^{0}$  and the effect of  $\mathsf{Stab}_{\epsilon}$  on them. The answer is evident in case  $\mathsf{Stab}_{\epsilon} = G$ , that  $G$ ,  $\mathcal{Z}^0$  are manifolds, and  $\mathcal{Z} = \mathcal{Z}^0\times G$ . Since  $\mathsf{ddim}$  coincides with topological dimension, and we immediately have
$$
d _ {0} = d - \dim (G).
$$
Intuitively, the dimensionality of  $G$  can be understood as eliminating degrees of freedom from  $\mathcal{Z}$ , and it is this effect that improves generalization by  $n^{-1/(d - \dim(G))} - n^{-1/d}$ .
We now state a simplified form of Theorem A.3, which is sufficient for proceeding further. Our results generalize [51, Thm. 3]: we allow for non-product structure, possibly infinite sets of transformations that may not form a group, and we relax the distance deformation hypotheses for the action on  $\mathcal{Z}$ .
Corollary 4.3 (of Thm. A.3). With the same notation as above, assume that for  $\mathsf{Stab}_{\epsilon}$  the transformations corresponding to the action of elements of  $\mathsf{Stab}_{\epsilon}$  satisfy the following for some  $L > 0$ , and for a choice of a set of representatives  $\mathcal{Z}_{\epsilon}^{0} \subset \mathcal{Z}$  of representatives of  $\mathsf{Stab}_{\epsilon}$ -orbits:
1. For all  $z_0, z_0' \in \mathcal{Z}_\epsilon^0$  and all  $g \in \operatorname{Stab}_\epsilon$  it holds that  $\mathsf{d}(z_0, z_0') \leq L \mathsf{d}(g \cdot z_0, g \cdot z_0')$ .
2. For all  $g, g' \in \mathsf{Stab}_{\epsilon}$  it holds that  $\mathsf{d}_G(g, g') \leq L \, \mathrm{dist}(g \cdot \mathcal{Z}_{\epsilon}^0, g' \cdot \mathcal{Z}_{\epsilon}^0)^6$ .
Then for each  $\delta > 0$  there holds  $\mathcal{N}(\mathcal{Z}_{\epsilon}^{0}, \delta) \leq \mathcal{N}(\mathcal{Z}, \delta / 2L) / \mathcal{N}(\mathrm{Stab}_{\epsilon}, \delta)$ .
To be able to quantify the precise generalization benefit we need a bound on the quantity  $\mathcal{N}(\mathsf{Stab}_{\epsilon},\delta)$ . For doing so, we assume that  $\mathsf{Stab}_{\epsilon}$  is a finite subset of a discrete group  $G$ , or is a positive measure subset of a compact group  $G$ . As before, let  $|G|$  and  $\mathsf{d}_G$  denote  $\sharp G$  and  $\mathsf{ddim}(G)$  respectively for finite  $G$ . Also, denote the Hausdorff measure and dimension by  $\mathsf{Vol}(G)$  and  $\mathsf{dim}(G)$  for compact metric groups. Note that the minimum separation  $\delta_G$  is zero for  $\mathsf{dim}(G) > 0$ . Then our covering bound can be expressed in the following umbrella statement:
$$
\text {A s s u m p t i o n :} \quad \mathcal {N} (\operatorname {S t a b} _ {\epsilon}, \delta) \gtrsim \max  \left\{1, \frac {| \operatorname {S t a b} _ {\epsilon} |}{(\max  \{\delta , \delta_ {G} \}) ^ {\mathrm {d} _ {G}}} \right\}. \tag {4.7}
$$
In order to compare the above to the precise equivariance case, we later use the factor
$$
\operatorname {D e n s} (\epsilon) := \frac {\left| \operatorname {S t a b} _ {\epsilon} \right|}{| G |} \in (0, 1 ],
$$
which measures the richness of  $\mathsf{Stab}_{\epsilon}$  compared to the whole ambient group  $G$ , in terms of error  $\epsilon$ . The fact that  $\mathsf{Dens}(\epsilon) > 0$  follows from our assumption that  $|\mathsf{Stab}_{\epsilon}| > 0$ . Furthermore,  $\mathsf{Dens}(\epsilon)$  is always an increasing function of  $\epsilon$ , as follows from the definition of  $\mathsf{Stab}_{\epsilon}$ . With these considerations on coverings, we can state a general result quantifying the generalization benefit, where if we set  $\epsilon = 0$ , we recover the case of exact group equivariance.
Theorem 4.4. Let  $\epsilon >0$  be fixed. Assume that for a given ambient group  $G$  the almost stabilizers  $\mathrm{Stab}_{\epsilon}$  satisfy assumption (4.7) and that the assumptions of Corollary 4.3 hold for some finite  $L = L_{\epsilon} > 0$ . Then with the notations of Proposition 4.1 we have with probability  $\geq 1 - \delta$
$$
\operatorname {G e n E r r} (\mathcal {F}, \{Z _ {i} \}, \mathcal {D}) \lesssim n ^ {- 1 / 2} \sqrt {\| \mathcal {F} \| _ {\infty} \log (2 / \delta)} + 2 \epsilon + \left(E _ {\epsilon}\right),
$$
$$
(E _ {\epsilon}) := \left\{ \begin{array}{l l} \frac {4 ^ {d _ {0}} d _ {0}}{d _ {0} - 2} \delta_ {G} ^ {- d _ {0} / 2 + 1} \left(\frac {(2 L _ {\epsilon}) ^ {d} D ^ {d}}{| \mathsf {S t a b} _ {\epsilon} | n}\right) ^ {1 / 2} & \text {i f G i s f i n i t e a n d (2 L _ {\epsilon}) ^ {d} D ^ {d} <   | \mathsf {S t a b} _ {\epsilon} | n   \delta_ {G} ^ {d _ {0}},} \\ \frac {4 ^ {d _ {0}} d _ {0}}{d _ {0} - 2} \left(\frac {(2 L _ {\epsilon}) ^ {d} D ^ {d}}{| \mathsf {S t a b} _ {\epsilon} | n}\right) ^ {1 / d _ {0}} & \text {e l s e}. \end{array} \right.
$$
The interpretation of the above terms is direct: the term  $2\epsilon$  is the effect of approximate equivariance, and the term  $(E_{\epsilon})$  includes the dependence on  $|\mathsf{Stab}_{\epsilon}|$  and thus is relevant to the study of partial equivariance. In general the Lipschitz bound  $L_{\epsilon}$  is increasing in  $\epsilon$  as well, since  $\mathsf{Stab}_{\epsilon}$  includes more elements of  $G$  as  $\epsilon$  increases, and we can expect that actions of elements generating higher equivariance error in general have higher oscillations.
5 Approximation error bounds under approximate data equivariance
In the introduction, we stated that we aim to validate the intuition that that model is best whose symmetry aligns with that of the data. So far we have only given bounds on the generalization error. Note that our bounds hold for any data distribution: in particular, the bounds are not affected by whether the data distribution is  $G$ -equivariant or not. However,
Here  $\mathsf{dist}(A,B)\coloneqq \min \{\mathsf{d}(a,b):a\in A,b\in B\}$  denotes the distance between sets, induced by  $d$
the fact that data may have fewer symmetries than the ones introduced in the model will have a controlled effect on worsening the approximation error, as described in this section.
However, controlling the approximation error has an additional actor in the fray to complicate matters: the distinguishing power of the loss function. Indeed, having a degenerate loss function with a large minimum set will not distinguish whether the model is fit to the data distribution or not. Thus, lowering the discrimination power of the loss has the same effect as increasing the function space for which we are testing approximation error.
Assuming at first that  $\widetilde{\mathcal{F}}$  is composed of  $G$ -equivariant functions, we compare its approximation error to the one of the set  $\mathcal{M}$  of all measurable functions  $m: \mathcal{X} \to \mathcal{Y}$ . Recall that  $f(Z) = \ell(\widetilde{f}(X), Y)$  is a positive function for all  $\widetilde{f} \in \widetilde{\mathcal{F}}$ , thus, denoting by  $\mathcal{M}' := \{F(x, y) = \ell(m(x), y), m \in \mathcal{M}\}$ , the approximation error gap can be defined and bounded as follows (where  $F^* \in \mathcal{M}'$  is a function realizing the first minimum in the second line below):
$$
\begin{array}{l} \operatorname {A p p G a p} (\mathcal {F}, \mathcal {D},) := \operatorname {A p p E r r} (\mathcal {F}, \mathcal {D}) - \operatorname {A p p E r r} (\mathcal {M} ^ {\prime}, \mathcal {D}): = \min  _ {f \in \mathcal {F}} \mathbb {E} [ f (Z) ] - \min  _ {F \in \mathcal {M} ^ {\prime}} \mathbb {E} [ F (Z) ] \\ \geq \min _ {F \in \mathcal {M} ^ {\prime}} \mathbb {E} [ \mathbb {E} _ {g} [ F (g \cdot Z) ] ] - \min _ {F \in \mathcal {M} ^ {\prime}} \mathbb {E} [ F (Z) ] \geq \mathbb {E} [ \mathbb {E} _ {g} [ F ^ {*} (g \cdot Z) ] ] - \mathbb {E} [ F ^ {*} (Z) ] \\ \geq \min _ {F \in \mathcal {M} ^ {\prime}} \mathbb {E} [ \mathbb {E} _ {g} [ F (g \cdot Z) ] - F (Z) ]. \\ \end{array}
$$
With applications to neural networks in mind, we will work with the following simplifying assumptions:
1. The loss function quantitatively detects whether the label is correct, i.e. we have  $\ell(y, y') = 0$  if  $y = y'$ , and  $\ell(y, y') \geq \varphi(\mathsf{d}_{\mathcal{Y}}(y, y'))$  for a strictly convex  $\varphi$  with  $\varphi(0) = 0$ , where  $\mathsf{d}_{\mathcal{Y}}$  is the distance on  $\mathcal{Y}$ . This assumption seems reasonable for use in applications, where objective functions with strict convexity properties are commonly used, as they help for the convergence of optimization algorithms.
2. The minimization across all measurable functions  $\widetilde{\mathcal{F}} = \mathcal{M}$  produces a zero-risk solution, i.e.  $\min_{F\in \mathcal{M}'}\mathbb{E}[F(Z)] = 0$  is achieved by a measurable function  $y^{*}: \mathcal{X}\to \mathcal{Y}$ . By assumption 1, this implies that  $\mathcal{D}$  is concentrated on the graph of  $y^{*}$ . This assumption corresponds to saying that the learning task is in principle deterministically solvable. Under this assumption, the task becomes to approximate to high accuracy the (unknown) solution  $y^{*}$ .
With the above hypotheses we have
$$
\operatorname {A p p G a p} (\mathcal {F}, \mathcal {D}) = \operatorname {A p p E r r} (\mathcal {F}, \mathcal {D}) \geq \min  _ {F \in \mathcal {M} ^ {\prime}} \mathbb {E} _ {Z} [ \mathbb {E} _ {g} [ F (g \cdot Z) ] ].
$$
As a slight simplification of Assumption 1 for ease of presentation, we will simply take  $\mathcal{Y} = \mathbb{R}^d$  and  $\ell(y, y') = \mathrm{d}_{\mathcal{Y}}(y, y')^2$  below. Note that Assumption 1 directly reduces to this case, as a strictly convex  $\varphi(t)$  with  $\varphi(0) = 0$  as in Assumption 1 is automatically bounded below by a multiple of  $t^2$ .
Now, with the aim to capture the mixed effects of approximate and partial equivariance, we introduce the following set of model classes. For  $\epsilon \geq 0, \lambda \in (0,1]$ :
$$
\mathcal {C} _ {\epsilon , \lambda} := \left\{\mathcal {F} \subset \mathcal {M} ^ {\prime}: \operatorname {D e n s} (\epsilon) = \frac {| \operatorname {S t a b} _ {\epsilon} (\mathcal {F}) |}{| G |} \geq \lambda \right\}.
$$
In order to establish some intuition, note that  $\mathsf{Stab}_{\epsilon}(\mathcal{F})$  is increasing in  $\epsilon$  and as a consequence  $\mathcal{C}_{\lambda,\epsilon}$  is also increasing in  $\epsilon$ . Directly from the definition one finds that  $\mathsf{Stab}_0(\mathcal{F})$  is necessarily a subgroup of  $G$ , and thus we allow  $\epsilon > 0$ , which allows more general sets of symmetries than just subgroups of  $G$ . The most interesting parameter in the above definition of  $\mathcal{C}_{\epsilon,\lambda}$  is  $\lambda$ , which actually bounds from below the "amount of prescribed symmetries" for our models. In the fully equivariant case  $\epsilon = 0, \lambda = 1$ , we have the simple description  $\mathcal{C}_{0,1} = \{\mathcal{F}: \mathcal{F} \subseteq (\mathcal{M}^{(G)})'\}$ , whose maximal element is  $(\mathcal{M}^{(G)})'$ . However in general  $\mathcal{C}_{\epsilon,\lambda}$  will not have a unique maximal element for  $\lambda < 1$ : this is due to the fact that two different elements  $\mathcal{F}_1 \neq \mathcal{F}_2 \in \mathcal{C}_{\epsilon,\lambda}$  may have incomparable stabilizers, so that  $|\mathsf{Stab}_{\epsilon}(\mathcal{F}_1) \setminus \mathsf{Stab}_{\epsilon}(\mathcal{F}_2)| > 0$  and  $|\mathsf{Stab}_{\epsilon}(\mathcal{F}_2) \setminus \mathsf{Stab}_{\epsilon}(\mathcal{F}_1)| > 0$  and thus  $\mathcal{F}_1 \cup \mathcal{F}_2 \notin \mathcal{C}_{\epsilon,\lambda}$ . Our main result towards our approximation error bound is the following.
Proposition 5.1. Assume that  $(\mathcal{V},\mathsf{d}_{\mathcal{V}})$  is a compact metric space and consider  $\ell (y,y^{\prime}) =$ $\mathtt{d}_{\mathcal{V}}(y,y^{\prime})^{2}$  . Fix  $\epsilon \geq 0$  . Then for each  $\lambda \in (0,1]$  there exists an element  $\mathcal{F}\in \mathcal{C}_{\epsilon ,\lambda}$  and a measurable selection  $\iota_{\epsilon}^{0}:\mathcal{X}\to \mathcal{X}_{\epsilon}^{0}$  of representatives of orbits of  $\mathcal{X}$  under the action of  $\operatorname {Stab}_\epsilon (\mathcal{F})$  , such that if  $X_{\epsilon}^{0},g$  are random variables obtained from  $X\sim \mathcal{D}$  by imposing  $X_{\epsilon}^{0}\coloneqq \iota_{\epsilon}^{0}\circ X$  and  $X = g\cdot X_{\epsilon}^{0}$  , then we have
$$
\mathsf {A p p E r r} (\mathcal {F}, \mathcal {D}) \leq \mathbb {E} _ {X _ {\epsilon} ^ {0}} \min  _ {y} \mathbb {E} _ {g | X _ {\epsilon} ^ {0}} \left[ \left(\max  \left\{\mathrm {d} _ {\mathcal {Y}} (g \cdot y,   y ^ {*} (g \cdot X _ {\epsilon} ^ {0})) - \epsilon ,   0 \right\}\right) ^ {2} \right].
$$
In the above, as  $\epsilon$  increases, we see a decrease in the approximation gap, since in this case model classes  $\mathcal{F}$  allow more freedom for approximating  $y^{*}$  more accurately. On the other hand, the effect of parameter  $\lambda$  is somewhat more subtle, and it has to do with the fact that as  $\lambda$  decreases,  $\operatorname{Stab}_{\epsilon}(\mathcal{F})$  for  $\mathcal{F} \in \mathcal{C}_{\epsilon,\lambda}$  can become smaller and the allowed oscillations of  $y^{*}(g \cdot X_{\epsilon}^{0})$  are more and more controlled.
Now to bound the approximation error above using the quantities that we have been working with, we will use the following version of an isodiametric inequality on metric groups  $G$ . As before we assume that  $G$  has Hausdorff dimension  $\dim(G) > 0$  which coincides with the doubling dimension, or is discrete. By  $|X|$  we denote the Hausdorff measure of  $X \subseteq G$  if  $\dim(G) > 0$  or the cardinality of  $X$  if  $G$  is finite. Then there exists an isodiametric constant  $C_G$  depending only on  $G$  such that for all  $X \subseteq G$  it holds that
$$
\frac {| X |}{| G |} \geq \lambda \Rightarrow \operatorname {d i a m} (X) \geq C _ {G} \lambda^ {1 / \operatorname {d d i m} (G)}. \tag {5.1}
$$
The above bound can be obtained directly from the doubling property of  $G$ , applied to a cover of  $X$  via a single ball of radius  $\text{diam}(X)$ , which can be refined to obtain better and better approximations of  $|X|$ . We can now control the bound from Proposition 5.1 as follows:
Theorem 5.2. Assume that  $(\mathcal{Y},\mathsf{d}_{\mathcal{Y}})$  is a compact metric space and  $\ell (y,y^{\prime}) = \mathsf{d}_{\mathcal{Y}}(y,y^{\prime})^{2}$ . Let  $G$  be a compact Lie group or a finite group, with distance  $\mathsf{d}_G$ . Let  $\mathsf{ddim}(G)$  be the doubling dimension of  $G$ , and assume that for all  $g\in G$  such that the action of  $g$  over  $\mathcal{X}$  is defined, it holds that  $\mathsf{d}(g\cdot x,g'\cdot x)\leq L'\mathsf{d}_G(g,g')$ . Then, there exists a constant  $C_G$  depending only on  $G$  such that for all  $\lambda \in (0,1]$  and  $\epsilon >0$ , there exists an element  $\mathcal{F}\in \mathcal{C}_{\epsilon ,\lambda}$  where for any data distribution  $\mathcal{D}$ ,  $y^{*}$  can be chosen Lipschitz with constant  $\mathrm{Lip}(y^{*})$ . We have
$$
\operatorname {A p p E r r} (\mathcal {F}, \mathcal {D}) \leq \left(\max  \left\{C _ {G} L ^ {\prime} (1 + \operatorname {L i p} (y ^ {*})) \lambda^ {1 / \mathrm {d d i m} (G)} - \epsilon , 0 \right\}\right) ^ {2}.
$$
We note that, given our proof strategy of Proposition 5.1, we conjecture that actually the above bound is sharp up to a constant factor:  $\min_{\mathcal{F} \in \mathcal{C}_{\epsilon, \lambda}} \mathsf{AppErr}(\mathcal{F}, \mathcal{D})$  may have a lower bound comparable to the above upper bound. The goal of the above result is to formalize the property that the approximation error guarantees of  $\mathcal{F}$  will tend to grow as the amount of imposed symmetries grow, as measured by  $\lambda$ .
6 Discussion: Imposing optimal equivariance
We have so far studied bounds on the generalization and the approximation errors, each of which take a very natural form in terms of properties of the space of orbit representatives (and thus the underlying  $G$ ). One of them depends on the data distribution, while the other doesn't. Using them we can thus state a result that quantifies the performance error, or gives the generalization-approximation trade-off, of a model based on the data distribution (and their respective symmetries). Define the performance error of a model class  $\mathcal{F} \in \mathcal{C}_{\epsilon,\lambda}$  over an i.i.d. sample  $\{Z_i\}_{i=1}^n$ ,  $Z_i \sim \mathcal{D}$  as
$$
\operatorname {P e r f E r r} \left(\mathcal {F}, \left\{Z _ {i} \right\}, \mathcal {D}\right) := \operatorname {G e n E r r} \left(\mathcal {F}, \left\{Z _ {i} \right\}, \mathcal {D}\right) + \operatorname {A p p E r r} \left(\mathcal {F}, \mathcal {D}\right).
$$
Combining Theorems 4.4 and 5.2, we get the following:
Theorem 6.1. Under the assumptions of Theorems 4.4 and 5.2, we have that, assuming all the function classes are restricted to functions with values in  $[-M, M]$ , then there exist
values  $C_1, C_2, C_3 >$  depending only on  $d_0, (2LD)^d / |G|, \delta_G$  and  $C = C_{G}L^{\prime}(1 + \mathrm{Lip}(y^{*}))$  such that with probability at least  $1 - \delta$ ,
$$
\mathsf {P e r f E r r} (\mathcal {F}, \{Z _ {i} \}, \mathcal {D}) \lesssim n ^ {- 1 / 2} \sqrt {M \log (2 / \delta)} + 2 \epsilon + (C \lambda^ {1 / \mathsf {d} _ {G}} - \epsilon) _ {+} ^ {2} + \left\{ \begin{array}{l l} C _ {1} \frac {1}{(n \lambda) ^ {1 / 2}} & \text {i f} n \lambda \geq C _ {3}, \\ C _ {2} \frac {1}{(n \lambda) ^ {1 / d _ {0}}} & \text {i f} n \lambda <   C _ {3}. \end{array} \right.
$$
Note that the above bounds are not informative as  $\lambda \rightarrow 0$  for the continuous group case, which corresponds to the second option in Theorem 6.1. This can be sourced back to the form of Assumption (4.7), in which the covering number on the left is always  $\geq 1$ , whereas in the continuous group case, the right-hand side may be arbitrarily small.
Thus the result is only relevant for  $\lambda$  away from 0. If we fix  $\epsilon = 0$  and we optimize over  $\lambda \in (0,1]$  in the above bound, we find a unique minimum  $\lambda^{*}$  for the error bound, specifying the optimum size of  $\mathsf{Stab}_{\epsilon}$  for the corresponding case (see the Appendix for more details). This validates the intuition stated at the onset that for the best model its symmetries must align with that of the data. As an illustration, for  $C,C_1,C_2 = 0.04$  and  $C_3 = 0.01$  with  $n = 1000000$ , the error trade-off is shown on the figure on the right.
Error versus lambda for large  $n$  for fixed values of  $C, C_1, C_2, C_3$ .
In this paper, we have provided general quantitative bounds showing that machine learning models with task-pertinent symmetries improve generalization. Our results do not require the symmetries to be a group and can work with partial/approximate equivariance. We also presented a general result which confirms the prevalent intuition that if the symmetry of a model is mis-specified w.r.t to the data symmetry, its performance will suffer. We now indicate some important future research directions, that correspond to limitations of our work:
- (Model-specific bounds) While in Theorem 6.1 we obtain the existence of an optimal amount  $\lambda^{*}$  without hopes to be sharp in such generality, we may expect that if we restrict to specific tasks, the bounds can become much tighter, and if so, the value  $\lambda^{*}$  of optimal amount of symmetries can become interpretable and give insights into the structure of the learning tasks.
- (T tighter bounds beyond classical PAC theory) For the sake of making the basic principles at work more transparent, we based our main result on long-known classical results. However, tighter data-dependent or probabilistic bounds would be very useful for getting a sharper value for the optimal value of symmetries  $\lambda^{*}$  via a stronger version of Theorem 6.1.
- (Beyond controlled doubling dimension) Our focus here is on groups  $G$  of controlled doubling dimension. This includes compact and nilpotent Lie groups, and discrete groups of polynomial growth, but the doubling dimension is not controlled for notable important cases, such as for the permutation group  $S_{n}$  (cf. section 3 of [17]) or for the group  $(\mathbb{Z}/2\mathbb{Z})^{n}$ . In order to cover these cases, it will be interesting to build analogues of our main result based on other group complexity bounds, beyond the doubling dimension case.
The authors thank Aryeh Kontorovich for readily answering questions (and supplying references) about Kolmogorov-Tikhomirov, and Bryn Elesedy for graciously answering queries and providing a copy of his dissertation before it appeared online. MP thanks the Centro Nacional de Inteligencia Artificial in Chile for support, as well as Fondecyt Regular grant number 1210462 entitled "Rigidity, stability and uniformity for large point configurations".
[66] Yinshuang Xu, Jiahui Lei, Edgar Dobriban, and Kostas Daniilidis. Unified fourier-based kernel and nonlinearity design for equivariant networks on homogeneous spaces. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 24596-24614. PMLR, 2022.
[67] Sicheng Zhu, Bang An, and Furong Huang. Understanding the generalization benefit of model invariance from a data perspective. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 4328-4341. Curran Associates, Inc., 2021.
[68] Xupeng Zhu, Dian Wang, Ondrej Biza, Guanang Su, Robin Walters, and Robert Platt. Sample efficient grasp learning using equivariant models. CoRR, abs/2202.09468, 2022.
A.1 Proofs of results from Section 4
A.1.1 Proof of Proposition 4.1
Proof. Reformulating (4.1) we find that with probability at least  $1 - \delta$  it holds that:
$$
\sup  _ {f \in \mathcal {F}} | (P - P _ {n}) f | \leq \mathcal {R} (\mathcal {F} _ {Z}) + \frac {\sqrt {2 \| \mathcal {F} \| _ {\infty} \log (2 / \delta)}}{\sqrt {n}}. \tag {A.1}
$$