SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mircea Petrache*
Shubhendu Trivedi†
The explicit incorporation of task-specific inductive biases through symmetry has emerged as a general design precept in the development of high-performance machine learning models. For example, group equivariant neural networks have demonstrated impressive performance across various domains and applications such as protein and drug design. A prevalent intuition about such models is that the integration of relevant symmetry results in enhanced generalization. Moreover, it is posited that when the data and/or the model may only exhibit approximate or partial symmetry, the optimal or best-performing model is one where the model symmetry aligns with the data symmetry. In this paper, we conduct a formal unified investigation of these intuitions. To begin, we present general quantitative bounds that demonstrate how models capturing task-specific symmetries lead to improved generalization. In fact, our results do not require the transformations to be finite or even form a group and can work with partial or approximate equivariance. Utilizing this quantification, we examine the more general question of model mis-specification i.e. when the model symmetries don't align with the data symmetries. We establish, for a given symmetry group, a quantitative comparison between the approximate/partial equivariance of the model and that of the data distribution, precisely connecting model equivariance error and data equivariance error. Our result delineates conditions under which the model equivariance error is optimal, thereby yielding the best-performing model for the given task and data. Our results are the most general results of their type in the literature.
It is a common intuition that machine learning models that explicitly incorporate task-specific symmetries tend to exhibit superior generalization. The rationale is simple: if aspects of the task remain unchanged or change predictably when subject to certain transformations, then, a model without knowledge of them would require additional training samples to learn to disregard them. While the general idea is by no means new in machine learning, it has experienced a revival in interest due to the remarkable successes of group equivariant convolutional neural networks (GCNNs) [14, 38, 29, 16, 13, 60, 8, 34, 24, 66]. Such networks hard-code equivariance to the action of an algebraic group on the inputs and have shown promise in a variety of domains [55, 5, 42, 2, 28, 62, 25, 52, 18, 61, 68], in particular in those where data is scarce, but exhibits clear symmetries. Further, the benefits of encoding symmetries in this manner extend beyond merely enhancing generalization. Indeed, in
*UC Chile, Fac. de Matemáticas, & Inst. de Ingeniería Matematica y Computacional, Av. Vicuña Mackenna 4860, Santiago, 6904441, Chile. mpetrache@mat.uc.cl.
$\dagger$ shubhendu@csail.mit.edu.
numerous applications in the physical sciences, neglecting to do so could lead to models producing unphysical outputs. However, the primary focus of most of the work in the area has been on estimating functions that are strictly equivariant under a group action. It has often been noted that this may impose an overly stringent constraint as real-world data seldom exhibits perfect symmetry. The focus on strict equivariance is partly due to convenience: the well-developed machinery of group representation theory and noncommutative harmonic analysis can easily be marshalled to design models that are exactly equivariant. A general prescriptive theory for such models also exists [29, 16]. In a certain sense, the basic architectural considerations of such models are fully characterized.
Recent research has begun to explore models that enforce only partial or approximate equivariance [40, 59, 23, 56]. Some of these works suggest interpolating between models that are exactly equivariant and those that are fully flexible, depending on the task and data. The motivation here is analogous to what we stated at the onset. If the data has a certain symmetry – whether exact, approximate or partial – and the model's symmetry does not match with it, its performance will suffer. We would expect that a model will perform best when its symmetry is correctly specified, that is, it aligns with the data symmetry.
Despite increased research interest in the area, the theoretical understanding of these intuitions remains unsatisfactory. Some recent work has started to study improved generalization for exactly equivariant/invariant models (see section 2). However, for the more general case when the data and/or model symmetries are only approximate or partial, a general treatment is completely missing. In this paper, we take a step towards addressing this gap. First, we show that a model enforcing task-specific invariance/equivalence exhibits better generalization. Our theoretical results subsume many existing results of a similar nature. Then we consider the question of model mis-specification under partial/approximate symmetries of the model and the data and prove a general result that formalizes and validates the intuition we articulated above.
We summarize below the main contributions of the paper:
- We present quantitative bounds, the most general to date, showing that machine learning models enforcing task-pertinent symmetries in an equivariant manner afford better generalization. In fact, our results do not require the set of transformations to be finite or even be a group. While our investigation was initially motivated by GCNNs, our results are not specific to them.
- Using the above quantification, we examine the more general setting when the data and/or model symmetries are approximate or partial and the model might be misspecified relative to the data symmetry. We rigorously formulate the relationship between model equivariance error and data equivariance error, teasing out the precise conditions when the model equivariance error is optimal, that is, provides the best generalization for given data. To the best of our knowledge, this is the most general result of its type in the literature.
As noted earlier, the use of symmetry to encode inductive biases is not a new idea in machine learning and is not particular to neural networks. One of the earliest explorations of this idea in the context of neural networks appears to be [46]. Interestingly, [46] originated in response to the group invariance theorems in Minsky & Papert's Perceptrons [36], and was the first paper in what later became a research program carried out by Shawe-Taylor & Wood, building more general symmetries into neural networks [63, 48, 49, 65, 64], which also included PAC style analysis [50]. While Shawe-Taylor & Wood came from a different perspective and already covered discrete groups [14, 38], modern GCNNs go beyond their work [29, 16, 13, 60, 8, 34, 24]. Besides, GCNNs [14] were originally proposed as generalizing the idea of the classical convolutional network [32], and seem to have been inspired by symmetry considerations from physics [15]. Outside neural networks, invariant/equivariant methods have also been proposed in the context of support vector machines [43, 57], general kernel methods [39, 26], and polynomial-based feature spaces [44].
On the theoretical side, the assertion that invariances enhance generalization can be found in many works, going back to [47]. It was argued by [1] that restricting a classifier to be invariant can not increase its VC dimension. Other examples include [3, 4, 51, 41]. The argument in these, particularly [51, 41] first characterizes the input space and then proceeds to demonstrate that certain data transformations shrink the input space for invariant models, simplifying the input and improving generalization. Some of our results generalize their results to the equivariant case and for more general transformations.
Taking a somewhat different tack, [22] showed a strict generalization benefit for equivariant linear models, showing that the generalization gap between a least squares model and its equivariant counterpart depends on the dimension of the space of anti-symmetric linear maps. This result was adapted to the kernel setting in [19]. [9] studied sample complexity benefits of invariance in a kernel ridge regression setting under a geometric stability assumption, and briefly discussed going beyond group equivariance. [53] expands upon [9] by characterizing minimax optimal rates for the convergence of population risk in a similar setting when the data resides on certain manifolds. [35] characterizes the benefits of invariance in overparameterized linear models, working with invariant random features projected onto high-dimensional polynomials. Benefits of related notions like data augmentation and invariant averaging are formally shown in [33, 12]. Using a standard method from the PAC-Bayesian literature [37] and working on the intertwiner space, [7] provide a PAC-Bayesian bound for equivariant neural networks. Some improved generalization bounds for transformation-invariant functions are proposed in [67], using the notion of an induced covering number. A work somewhat closer to some of our contributions in this paper is [20], which gives PAC bounds for (exact) group equivariant models. However, we note that the proof of the main theorem in [20] has an error (see the Appendix for a discussion), although the error seems to be corrected in the author's dissertation [21]. An analysis that somewhat overlaps [20], but goes beyond simply providing worst-case bounds was carried out by [45].
Finally, recent empirical work on approximate and partial equivariance includes [40, 59, 23, 56]. These make the case that enforcing strict equivariance, if misspecified, can harm performance, arguing for more flexible models that can learn the suitable degree of equivariance from data. However, there is no theoretical work that formalizes the underlying intuition.
In this section, we introduce basic notation and formalize some key notions that will be crucial to ground the rest of our exposition.
3.1 Learning with equivariant models
Learning problem. To begin, we first describe the general learning problem which we adapt to the equivariant case. Let's say we have an input space  $\mathcal{X}$  and an output space  $\mathcal{Y}$ , and assume that the pairs  $Z = (X,Y)\in \mathcal{X}\times \mathcal{Y}$  are random variables having distribution  $\mathcal{D}$ . Suppose we observe a sequence of  $n$  i.i.d pairs  $Z_{i} = (X_{i},Y_{i})\sim \mathcal{D}$ , and want to learn a function  $\tilde{f}:\mathcal{X}\to \mathcal{Y}$  that predicts  $Y$  given some  $X$ . We will consider that  $\tilde{f}$  belongs to a class of functions  $\widetilde{\mathcal{F}}\subset \{\tilde{f}:\mathcal{X}\to \mathcal{Y}\}$  and that we work with a loss function  $\ell :\mathcal{X}\times \mathcal{Y}\rightarrow [0,\infty)$ . To fully specify the learning problem we also need a loss class, comprising of the functions  $f(x,y)\coloneqq \ell (\tilde{f} (x),y):\mathcal{X}\times \mathcal{Y}\rightarrow [0,\infty)$ . Using these notions, we can define:
$$
\mathcal {F} := \left\{f (x, y) := \ell (\widetilde {f} (x), y): \tilde {f} \in \widetilde {\mathcal {F}} \right\}, \quad P f := \mathbb {E} [ f (X, Y) ], \quad P _ {n} f := \frac {1}{n} \sum_ {i = 1} ^ {n} f (X _ {i}, Y _ {i}).
$$
The quantities  $Pf$  and  $P_{n}f$  are known as the risk and the empirical risk associated with  $\widetilde{f}$ , respectively. In practice we have access to  $P_{n}f$ , and we estimate  $Pf$  by it. One of the chief quantities of interest in such a setup is the worst-case error of this approximation on a sample  $\{Z_{i}\} = \{Z_{i}\}_{i=1}^{n}$ , that is, the generalization error:
$$
\operatorname {G e n E r r} (\mathcal {F}, \{Z _ {i} \}, \mathcal {D}) := \sup  _ {f \in \mathcal {F}} (P f - P _ {n} f).
$$
Group actions. We are interested in the setting where a group  $G$  acts by measurable bijective transformations over  $\mathcal{X}$  and  $\mathcal{Y}$ , which can also be seen as a  $G$ -action over  $Z$ , denoted
by  $g \cdot Z = (g \cdot X, g \cdot Y)$ , where  $g \in G$ . Such actions can also be applied to points  $(x, \widetilde{f}(x))$  in the graph of functions  $\widetilde{f}$ , transforming any  $\widetilde{f} \in \widetilde{\mathcal{F}}$  into  $g \cdot \widetilde{f} : x \mapsto g^{-1} \cdot \widetilde{f}(g \cdot x)$ . The set  $G \cdot \widetilde{f} := \{g \cdot \widetilde{f} : g \in G\}$  forms the orbit of  $\widetilde{f}$  under the action of  $G$ . Note that although we treat  $G$  as an abstract group throughout, we will sometimes abuse notation and use the same letter to also refer to the representations of  $G$  as transformations over the spaces in which  $X, Y, Z, \widetilde{f}, f$  live.
Equivariant models. We are now in a position to specify the learning setup with equivariant models. We say that  $\widetilde{f} \in \widetilde{\mathcal{F}}$  is equivariant, if  $G \cdot \widetilde{f} = \{\widetilde{f}\}$ . Further, we say that the hypothesis class is  $G$ -equivariant if all  $\widetilde{f} \in \widetilde{\mathcal{F}}$  are equivariant. Note that if the  $G$ -action over  $Y$  is trivial, that is, if we have  $g \cdot Y = Y$  for all  $g \in G, y \in Y$ , then  $\widetilde{f}$  being equivariant reduces to requiring  $\widetilde{f}(g \cdot x) = \widetilde{f}(x)$  for all  $g, x$ . In such a case the model  $\widetilde{f}$  is said to be invariant. Also note that  $\widetilde{f}(x)$  being equivariant corresponds to  $f(x, y) = \ell(\widetilde{f}(x), y)$  being invariant under some suitable product  $G$ -action over  $\mathcal{X} \times \mathcal{Y}$ .
Averaging over a set of transformations. Let  $G$  be a set of transformations of  $\mathcal{Z}$ . The  $G$ -averaged generalization error for arbitrary  $\mathcal{F}$  by taking averages  $\mathbb{E}_g$  with respect to the natural probability measure  $\mathbb{P}_G^3$  on  $G$  is as follows:
$$
\operatorname {G e n E r r} ^ {(G)} (\mathcal {F}, \{Z _ {i} \}, \mathcal {D}) := \sup  _ {f \in \mathcal {F}} \left(\mathbb {E} _ {g} \mathbb {E} _ {Z} [ f (g \cdot Z) ] - \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {g} f (g \cdot Z _ {i})\right),
$$
Note that the above equals  $\mathsf{GenErr}(\mathcal{F},\{Z_i\},\mathcal{D})$  if  $\mathcal{F}$  is composed of invariant functions.
Data augmentation/Orbit averaging. Following our discussion above, it might be instructive to consider the case of data augmentation. Note that a data distribution which is invariant under some  $G$ -action can be created from any  $\mathcal{D}$  by averaging over  $G$ -orbits: we take  $g$  to be a random variable uniformly distributed over a compact set of transformations  $G$ . Then let  $\mathcal{D}^{(G)} \coloneqq \frac{1}{|G|} \int_{G} g \cdot \mathcal{D}$ . This is the distribution of the dataset obtained from  $\mathcal{D}$  under  $G$ -augmentation. Then we have by change of variables:
$$
\operatorname {G e n E r r} ^ {(G)} \left(\mathcal {F}, \left\{Z _ {i} \right\}, \mathcal {D}\right) = \operatorname {G e n E r r} \left(\mathcal {F}, \left\{\mathrm {E} _ {g} [ g \cdot Z _ {i} ] \right\}, \mathcal {D} ^ {(G)}\right). \tag {3.1}
$$
Note that the notions of orbit and averaging used above do not rely on the property that  $G$  is a group in order to be well-defined.
3.2 Partial and Approximate Equivalence
Since our analysis treats the general case where the model and/or data symmetries are not strict, we now state some basic attendant notions.
Error to equivariance function. We first need a measure to quantify how far off we are from perfect equivariance. To do so, we can define a function in terms of  $f(x,y)$  as:
$$
\mathbf {e e}: \mathcal {F} \times G \rightarrow [ 0, + \infty), \quad \mathbf {e e} (f, g) = \| g \cdot f - f \| _ {\infty}.
$$
The crucial property here is that  $\mathbf{ee}(f,g) = 0$  iff  $g\cdot f = f$ . Note that one could use other distances or discrepancies (e.g. like the  $\ell_2$  norm) directly in terms of  $\widetilde{f}$ , using which we can measure the error between  $g\cdot \widetilde{f}$  and  $\widetilde{f}$ .
More general sets of transformations. We can use  $\mathbf{ee}(\cdot)$  to precisely specify the notions of partial and approximate equivariance that we work with. We consider the general case where the underlying transformations on  $(X,Y)$  still have a common label set  $G$  and are assumed to be bijective over  $\mathcal{X}$  and  $\mathcal{Y}$ , but don't necessarily form a group. For  $\epsilon > 0$  and  $f \in \mathcal{F}$  we consider the  $\epsilon$ -stabilizer
$$
\operatorname {S t a b} _ {\epsilon} (f) := \{g \in G: \operatorname {e e} (f, g) \leq \epsilon \}.
$$
It is fairly easy to see that this subset is automatically a group for  $\epsilon = 0$ , but most often will not be when  $\epsilon > 0$ . The latter corresponds to partial symmetry and thus to the setting
<sup>3</sup>We take  $\mathbb{P}_G$  to be the uniform measure over  $G$  if  $G$  is finite, and the normalization of the Haar measure of  $\overline{G}$  to  $G$  if  $G \subset \overline{G}$  is a positive measure subset of an ambient locally compact group  $\overline{G}$ .
of partial equivariance. On the other hand, if we consider specific hypotheses such that for some  $\epsilon > 0$  we have  $\operatorname{Stab}_{\epsilon}(f) = G$  for all  $f \in \mathcal{F}$ , then we are working in the setting of approximate equivariance [59, 56].
Conditions on  $G$ . We conclude this section with a remark on the nature of  $G$  we consider in our paper. In our bounds, we first restrict to finite  $G$ . However, this setting can be extended to compact groups which are doubling with respect to a metric that respects the composition operation. That is, a distance  $\mathsf{d}_G$  over  $G$ , such that  $\mathsf{d}_G(gh,g'h) = d(g,g')$ . The main examples we have in mind are compact Lie groups such as  $S^1, SO(n), O(n)$ , and their finite subgroups like  $\mathbb{Z} / N\mathbb{Z}$ , as well as their quotients and products.
4 PAC bounds for  $G$ -equivariant hypotheses
In this section, we study the generalization properties of equivariant models. More specifically, we derive PAC bounds for both exact and approximately G-equivariant hypotheses. But first, we present some preliminaries related to PAC bounds in what follows.
PAC learning bounds.
We follow the standard approach (such as that outlined in [10]). We start with the following concentration bound, where  $\mathcal{F}$  is composed of functions with range  $[-M/2, M/2]$ :
$$
\mathbb {P} \left[ \sup  _ {\mathcal {F}} | (P - P _ {n}) f | \geq \mathcal {R} \left(\mathcal {F} _ {Z}\right) + \epsilon \right] \leq 2 \exp \left(- \frac {\epsilon^ {2} n}{M}\right), \tag {4.1}
$$
where  $\mathcal{R}_n(\mathcal{F})$  denotes the Rademacher complexity, which is given by
$$
\mathcal {R} \left(\mathcal {F} _ {Z}\right) := \mathbb {E} _ {\sigma} \sup  _ {\mathcal {F}} \frac {1}{n} \sum_ {i = 1} ^ {n} \sigma_ {i} f \left(Z _ {i}\right),
$$
and where  $Z_{1},\ldots ,Z_{n}\sim \mathcal{D}$  are  $n$  i.i.d. samples and  $\sigma = (\sigma_1,\dots,\sigma_n)$  denotes the so-called Rademacher variable, which is a uniformly distributed random variable over  $\{-1,1\} ^n$  independent of the  $Z_{i}$ 's.  $\mathcal{R}(\mathcal{F}_Z)$  can be bounded using a classical technique for bounding the expected suprema of random processes indexed by the elements of a metric space, variously called the Dudley entropy integral or the chaining bound. The result below was proven in [45, Lemma 3] (also see slightly stronger results in [58, Thm. 17] or [6, Thm. 1.1])
$$
\mathcal {R} \left(\mathcal {F} _ {Z}\right) \leq 4 \inf  _ {\alpha > 0} \left(\alpha + \frac {3}{\sqrt {n}} \int_ {\alpha} ^ {\operatorname {d i a m} (\mathcal {F})} \sqrt {\ln \mathcal {N} \left(\mathcal {F} , t , \| \cdot \| _ {\infty}\right)} d t\right). \tag {4.2}
$$
Recall that for a metric space  $(X, \mathsf{d}_X)$ , the covering number  $\mathcal{N}(X, \epsilon, \mathsf{d}_X)$  is the smallest number of balls of radius  $\epsilon$  required to cover  $X$ . In (4.2) we use the cover numbers of  $\mathcal{F}$  in supremum distance, i.e. the distance between two functions  $f, g \in \mathcal{F}$ ,  $\| f - g \|_{\infty} \coloneqq \sup_{z \in \mathcal{Z}} |f(z) - g(z)|$ . It is known that by rescaling the fundamental estimate due to Kolmogorov-Tikhomirov [54, eq. 238, with  $s = 1$ , and eq. 1], and under the mild assumption that  $\mathcal{F}$  is composed of 1-Lipschitz<sup>4</sup> functions on  $\mathcal{Z}$  with values in an interval  $[-M/2, M/2]$ , for a centralizable<sup>5</sup> metric space  $\mathcal{Z}$ , the following holds
$$
\mathcal {N} (\mathcal {Z}, 2 \epsilon) \leq \log_ {2} \mathcal {N} (\mathcal {F}, \epsilon , \| \cdot \| _ {\infty}) \leq \log_ {2} \left(\frac {M}{\epsilon} + 1\right) + \mathcal {N} (\mathcal {Z}, \epsilon / 2). \tag {4.3}
$$
Before we start stating our results, we need to define a few more notions:
Doubling and Hausdorff dimensions. If in a metric space  $(X, \mathsf{d}_X)$ , every ball of radius  $R$  can be covered by at most  $\lambda$  balls of radius  $R/2$ , the space has doubling dimension  $\mathrm{ddim}(X) = \log_2 \lambda$ . The doubling dimension coincides with the usual notion of (Hausdorff) dimension  $\dim X$ , i.e.  $\mathrm{ddim}X = \dim X$ , in case  $X$  is a compact manifold with bounded injectivity radius, in particular it equals  $d$  if  $X \subset \mathbb{R}^D$  is a regular  $d$ -dimensional submanifold or if  $D = d$  and  $X$  is a regular open subset such as a ball or a cube.
4All of our formulas generalize to classes of  $\ell$ -Lipschitz functions for general  $\ell > 0$ : the change amounts to applying suitable rescalings, see the Appendix for more details.
This mild condition signifies that for any open set  $U$  of diameter at most  $2r$  there exists a point  $x^0$  so that  $U$  is contained in  $B(x^0, r)$ .
Discrete metric spaces. A metric space  $(X, \mathsf{d}_X)$  is discrete if it has strictly positive minimum separation distance  $\delta_X \coloneqq \min_{x \neq x' \in X} \mathsf{d}_X(x, x') > 0$ . We then have
$$
\mathcal {N} (X, \epsilon) = \mathcal {N} (X, \min  \{\epsilon , \delta_ {X} \}), \tag {4.4}
$$
which is especially relevant for finite groups  $G$  endowed with the word distance (i.e.  $\mathsf{d}_G(g, g')$ , which is the minimum number of generators required to express  $g^{-1}g'$ ), for which  $\delta_G = 1$ . Now, note that as a straightforward consequence of the definition, there exists a universal  $c > 0$  such that[30, 31]:
$$
\mathcal {N} (\mathcal {Z}, \epsilon) \leq \left(\frac {2 \operatorname {d i a m} (\mathcal {Z})}{\epsilon}\right) ^ {\operatorname {d d i m} (\mathcal {Z})}. \tag {4.5}
$$
and by (4.3) we get the simplified bound (where implicit constants are universal):
$$
\ln \mathcal {N} (\mathcal {F}, \epsilon , \| \cdot \| _ {\infty}) \lesssim \left(\frac {4 \operatorname {d i a m} (\mathcal {Z})}{\epsilon}\right) ^ {\operatorname {d d i m} (\mathcal {Z})}. \tag {4.6}
$$
By all the above considerations, we can bound the generalization error as follows:
Proposition 4.1. Assume that  $d = \mathbb{ddim}(\mathcal{Z}) > 2$ , and  $0 < \delta < 1/2$ , and let  $D := \mathrm{diam}(\mathcal{Z})$ . Then for any probability distribution  $\mathcal{D}$  of data over  $\mathcal{Z}$ , with notation as in the beginning of Section 4, the following holds with probability at least  $1 - \delta$ :
$$
\operatorname {G e n E r r} (\mathcal {F}, \{Z _ {i} \}, \mathcal {D}) \lesssim \frac {4 ^ {d} d}{d - 2} \left(\frac {D ^ {d}}{n}\right) ^ {1 / d} + n ^ {- 1 / 2} \sqrt {\| \mathcal {F} \| _ {\infty} \log (2 / \delta)},
$$
the implicit constant is independent of  $\delta, n$ ; only depending on  $\mathcal{Z}$  through the constants in (4.5).
Due to space constraints, all our proofs are relegated to the Appendix. With the above background, we now first show generalization bounds under the more general notions of equivariance we have discussed.
4.1 Generalization bounds improvement under partial or approximate equivariance
In this section, we prove generalization error bounds with the notations and definitions from Section 3.2. We consider the following sets of transformations: