SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Behrooz Tahmasebi
bzt@mit.edu
MIT CSAIL and TU Munich
stefje@mit.edu
In practice, encoding invariances into models improves sample complexity. In this work, we study this phenomenon from a theoretical perspective. In particular, we provide minimax optimal rates for kernel ridge regression on compact manifolds, with a target function that is invariant to a group action on the manifold. Our results hold for any smooth compact Lie group action, even groups of positive dimension. For a finite group, the gain effectively multiplies the number of samples by the group size. For groups of positive dimension, the gain is observed by a reduction in the manifold's dimension, in addition to a factor proportional to the volume of the quotient space. Our proof takes the viewpoint of differential geometry, in contrast to the more common strategy of using invariant polynomials. This new geometric viewpoint on learning with invariances may be of independent interest.
In a broad range of applications, including machine learning for physics, molecular biology, point clouds, and social networks, the underlying learning problems are invariant with respect to a group action. The invariances are observed widely in practice, for instance, in the study of high energy particle physics [21, 32], galaxies [23, 15, 2], and also molecular datasets [1, 59, 36] (see [67] for a survey). In learning with invariances, one aims to develop powerful architectures that exploit the problem's invariance structure as much as possible. An essential question is thus: what are the fundamental benefits of model invariance, e.g., in terms of sample complexity?
Several architectures for learning with invariances have been proposed for various types of data and invariances, including DeepSet [70] for sets, Convolutional Neural Networks (CNNs) [28], PointNet [53, 54] for point clouds with permutation invariance, tensor field neural networks [62] for point clouds with rotations, translations, and permutations symmetries, Graph Neural Networks (GNNs) [58], and SignNet and BasisNet [37] for spectral data. Other works study invariance with respect to the orthogonal group [63], and invariant and equivariant GNNs [42]. These architectures are to exploit the invariance of data as much as possible, and are invariant/equivariant by design.
In fixed dimensions, one common feature of many invariant models, including those discussed above, is that the data lie on a compact manifold (not necessarily a sphere, e.g., the Stiefel manifold for spectral data), and are invariant with respect to a group action on that manifold. Thus, characterizing the theoretical gain of invariances corresponds to studying the gain of learning under group actions on manifolds. Adopting this view, in this paper, we answer the question: how much gain in sample complexity is achievable by encoding invariances? As this problem is algorithm and model dependent, it is hard to address in general. A focused version of the problem, but still interesting, is to study this sample complexity gain in kernel-based algorithms, which is what we address here. As neural networks in certain regimes behave like kernels (for example, the Neural Tangent Kernel (NTK) [26, 34]), the results on kernels should be understood as relevant to a range of models.
Formally, we consider the Kernel Ridge Regression (KRR) problem with i.i.d. data on a compact manifold  $\mathcal{M}$ . The target function lies in a Reproducing Kernel Hilbert space (RKHS) of Sobolev functions  $\mathcal{H}^s (\mathcal{M})$ ,  $s\geq 0$ . In addition, the target function is invariant to the action of an arbitrary Lie group  $G$  on the manifold. We aim to quantify: by varying the group  $G$ , how does the sample complexity change, and what is the precise gain as  $G$  grows?
Main results. Our main results characterize minimax optimal rates for the convergence of the (excess) population risk (generalization error) of KRR with invariances. More precisely, for the Sobolev kernel, the most commonly studied case of kernel regression, we prove that a (excess) population risk (generalization error)  $\propto \left(\frac{\sigma^2\mathrm{vol}(\mathcal{M} / G)}{n}\right)^{s / (s + d / 2)}$  is both achievable and minimax optimal, where  $\sigma^2$  is the variance of the observation noise,  $\mathrm{vol}(\mathcal{M} / G)$  is the volume<sup>1</sup> of the corresponding quotient space, and  $d$  is the effective dimension of the quotient space (see Section 4 for a precise definition). This result shows a reduction in sample complexity in two intuitive ways: (1) scaling the effective number of samples, and (2) reducing dimension and hence exponent. First, for finite groups, the factor  $\mathrm{vol}(\mathcal{M} / G)$  reduces to  $\mathrm{vol}(\mathcal{M}) / |G|$ , and can hence be interpreted as scaling the effective number of samples by the size of the group. That is, each data point conveys the information of  $|G|$  data points due to the invariance. Second, and importantly, the parameter  $d$  in the exponent can generally be much smaller than  $\dim (\mathcal{M})$ , which would be the correspondent of  $d$  in the non-invariant case. In the best case,  $d = \dim (\mathcal{M}) - \dim (G)$ , where  $\dim (G)$  is the dimension of the Lie group  $G$ . Hence, the second gain shows a gain in the dimensionality of the space, and hence in the exponent.
Our results generalize and greatly expand previous results by Bietti et al. [5], which only apply to finite groups and isometric actions and are valid only on spheres. In contrast, we derive optimal rates for all compact manifolds and smooth compact Lie group actions (not necessarily isometric), including groups of positive dimension. In particular, the reduction in dimension applies to infinite groups, since for finite groups  $\dim(G) = 0$ . Hence, our results reveal a new perspective on the reduction in sample complexity that was not possible with previous assumptions. Our rates are consistent with the classical results for learning in Sobolev spaces on manifolds without invariances [24]. To illustrate our general results, in Section 5, we make them explicit for kernel counterparts of popular invariant models, such as DeepSets, GNNs, PointNet, and SignNet.
Even though our theoretical results look intuitively reasonable, the proof is challenging. We study the space of invariant functions as a function space on the quotient space  $\mathcal{M} / G$ . To bound its complexity, we develop a dimension counting theorem for functions on the quotient space, which is at the heart of our analysis and of independent interest. The difficulty is that  $\mathcal{M} / G$  is not always a manifold. Moreover, it may exhibit non-trivial boundaries that require boundary conditions to study function spaces. Different boundary conditions can lead to very different function spaces, and a priori the appropriate choice for the invariant functions is unclear. We prove that smooth invariant functions on  $\mathcal{M}$  satisfy the Neumann boundary condition on the (potential) boundaries of the quotient space, thus characterizing exactly the space of invariant functions.
The ideas behind the proof are of potential independent interest: we provide a differential geometric viewpoint of the class of functions defined on manifolds and study group actions on manifolds from this perspective. This stands in contrast to the classical strategy of using polynomials generating the class of functions [46, 5], which is restricted to spheres. To the best of our knowledge, the tools used in this paper are new to the literature on learning with invariances.
In short, in this paper we make the following contributions:
- We characterize the exact sample complexity gain from invariances for kernel regression on compact manifolds for an arbitrary Lie group action. Our results reveal two ways to reduce sample complexity, including a new reduction in dimensionality that was not obtainable with assumptions in prior work.
- Our proof analyzes invariant functions as a function space on the quotient space; this differential geometric perspective and our new dimension counting theorem, which is at the heart of our analysis, may be of independent interest.
<sup>1</sup>The quotient space is not a manifold, but one can still define a notion of volume for it; see Section 4.
The perhaps closest related work to this paper is [5], which considers the same setup for finite isometric actions on spheres. We generalize their result in several aspects: the group actions are not necessarily finite or isometric, and the compact manifold is arbitrary (including compact submanifolds of  $\mathbb{R}^d$ ), allowing to observe a new axis of complexity gain. Mei et al. [46] consider invariances for random features and kernels, but in a different scaling/regime; thus, theirs are not comparable to our results. For density estimation on manifolds, optimal rates are given in [24], which are consistent with our theory. McRae et al. [45] show non-asymptotic sample complexity bounds for regression on manifolds. A similar technique was recently applied in [40], but for a very different setting of covariate shifts.
The generalization benefits for invariant classifiers are observed in the most basic setup in [61], and for linear invariant/equivariant networks in [20, 19]. Some works propose covering ideas to measure the generalization benefit of invariant models [71], while others use properties of the quotient space [57, 50]. It is known that structured data exhibit certain gains for localized classifiers [14]. Sample complexity gains are also observed for CNN on images [17, 35]. Wang et al. [66] incorporate more symmetry in CNNs to improve generalization.
Many works introduce models for learning with invariances for various data types; in addition to those mentioned in the introduction, there are, e.g., group invariant scattering models [41, 9]. A probabilistic viewpoint of invariant functions [6] and a functional perspective [72] also exist in the literature. The connection between group invariances and data augmentation is addressed in [13, 39].
Universal expressiveness has been studied for settings like rotation equivariant point clouds [18], sets with symmetric elements [44], permutation invariant/equivariant functions [56], invariant neural networks [55, 69, 43], and graph neural networks [68, 49]. Lawrence et al. [30] study the implicit bias of linear equivariant networks. For surveys on invariant/equivariant neural networks, see [22, 8].
Consider a smooth connected compact boundaryless $^2$ $\dim (\mathcal{M})$ -dimensional (Riemannian) manifold  $(\mathcal{M},g)$ , where  $g$  is the Riemannian metric. Let  $G$  denote an arbitrary compact Lie group of dimension  $\dim(G)$  (i.e., a group with a smooth manifold structure), and assume that  $G$  acts smoothly on the manifold  $(\mathcal{M},g)$ ; this means that each  $\tau \in G$  corresponds to a diffeomorphism  $\tau : \mathcal{M} \to \mathcal{M}$ , i.e., a smooth bijection. Without loss of generality, we can assume that  $G$  acts isometrically on  $(\mathcal{M},g)$ , i.e.,  $G$  is a Lie subgroup of the isometry group  $\mathrm{ISO}(\mathcal{M},g)$ . To see why this is not restrictive, given a base metric  $g$ , consider a new metric  $\tilde{g} = \mu_G(\tau^* g)$ , where  $\mu_G$  is the left-invariant Haar (uniform) measure of  $G$ , and  $\tau^* g$  is the pullback of the metric  $g$  by  $\tau$ . Under the new metric,  $G$  acts isometrically on  $(\mathcal{M},\tilde{g})$ . We review basic facts about manifolds and their isometry groups in Appendix A.1 and Appendix A.2.
We are given a dataset  $\mathcal{S} = \{(x_i, y_i) : i = 1, 2, \dots, n\} \subseteq (\mathcal{M} \times \mathbb{R})^n$  of  $n$  labeled samples, where  $x_i \sim_{\mathrm{i.i.d.}} \mu$ , for the uniform (Borel) probability measure  $d\mu(x) := \frac{1}{\operatorname{vol}(\mathcal{M})} d\operatorname{vol}_g(x)$ . Here,  $d\operatorname{vol}_g(x)$  denotes the volume element of the manifold defined using the Riemannian metric  $g$ . We assume the uniform sampling for simplicity; our results hold for non-uniform cases, too. The hypothesis class is a set  $\mathcal{F} \subseteq L_{\mathrm{inv}}^2(\mathcal{M}, G) \subseteq L^2(\mathcal{M})$  including only  $G$ -invariant square-integrable functions on the manifold, i.e., those  $f \in L^2(\mathcal{M})$  satisfying  $f(\tau(x)) = f(x)$  for all  $\tau \in G$ . We assume that there exists a function  $f^\star \in \mathcal{F}$  such that  $y_i = f^\star(x_i) + \epsilon_i$  for each  $(x_i, y_i) \in S$ , where  $\epsilon_i$ 's are conditionally zero-mean random variables with variance  $\sigma^2$ , i.e.,  $\mathbb{E}[\epsilon_i | x_i] = 0$  and  $\mathbb{E}[\epsilon_i^2 | x_i] \leq \sigma^2$ .
Let  $K: \mathcal{M} \times \mathcal{M}$  denote a continuous positive-definite symmetric (PDS) kernel on the manifold  $\mathcal{M}$ , and let  $\mathcal{H} \subseteq L^2(\mathcal{M})$  denote its Reproducing Kernel Hilbert Space (RKHS). The kernel  $K$  is called
2Although the results in this paper can be easily extended to manifolds with boundaries, for simplicity, we focus on the boundaryless case.
$G$ -invariant  $^3$  if and only if for all  $x,y\in \mathcal{M}$
$$
K (x, y) = K (\tau (x), \tau^ {\prime} (y)), \tag {1}
$$
for any  $\tau, \tau' \in G$ . In other words,  $K(x, y) = K([x], [y])$ , where  $[x] := \{\tau(x) : \tau \in G\}$  is the orbit of the group action that includes  $x$ .
The Kernel Ridge Regression (KRR) problem on the data  $S$  with a  $G$ -invariant kernel  $K$  asks for the function  $\hat{f}$  that minimizes
$$
\hat {f} := \underset {f \in \mathcal {H}} {\arg \min } \left\{\hat {\mathcal {R}} (f) := \frac {1}{n} \sum_ {i = 1} ^ {n} \left(y _ {i} - f \left(x _ {i}\right)\right) ^ {2} + \eta \| f \| _ {\mathcal {H}} ^ {2} \right\}. \tag {2}
$$
By the representative theorem [48], the optimal solution  $\hat{f} \in \mathcal{H}$  is of the form  $\hat{f} = \sum_{i=1}^{n} a_i K(x_i, \cdot)$  for a weight vector  $\mathbf{a} \in \mathbb{R}^n$ . The objective function  $\hat{\mathcal{R}}(\hat{f})$  can thus be written as
$$
\hat {\mathcal {R}} (\hat {f}) = \frac {1}{n} \| \mathbf {y} - \mathbf {K a} \| _ {2} ^ {2} + \eta \mathbf {a} ^ {T} \mathbf {K a}, \tag {3}
$$
where  $\mathbf{y} = (y_{1}, y_{2}, \ldots, y_{n})^{n}$  and  $\mathbf{K} = \{K(x_{i}, x_{j})\}_{i,j=1}^{n}$  is the Gram matrix. This gives the closed form solution  $\mathbf{a} = (\mathbf{K} + n\eta I)^{-1}\mathbf{y}$ . Using the population risk  $\mathcal{R}(f) := \mathbb{E}_{x \sim \mu}[(y - f(x))^{2}]$ , the effective ridge regression estimator is defined as
$$
\hat {f} _ {\text {e f f}} := \underset {f \in \mathcal {H}} {\arg \min } \left\{\mathcal {R} (f) + \eta \| f \| _ {\mathcal {H}} ^ {2} \right\}. \tag {4}
$$
This paper focuses on the RKHS of Sobolev functions,  $\mathcal{H} = \mathcal{H}_{\mathrm{inv}}^s (\mathcal{M}) = \mathcal{H}^s (\mathcal{M})\cap L_{\mathrm{inv}}^2 (\mathcal{M},G)$ $s\geq 0$  . This includes all functions having square-integrable derivatives up to order  $s$  . Note that  $\mathcal{H}^s (\mathcal{M})$  includes only continuous functions when  $s > \dim (\mathcal{M}) / 2$  . Moreover, it contains only continuously differentiable functions up to order  $k$  when  $s > \dim (\mathcal{M}) / 2 + k$  (Appendix A.10). Note that  $\mathcal{H}^s (\mathcal{M})$  is an RKHS if and only if  $s > \dim (\mathcal{M}) / 2$
Our first theorem provides an upper bound on the excess population risk, or the generalization error, of KRR with invariances.
Theorem 4.1 (Convergence rate of KRR with invariances). Consider KRR with invariances with respect to a Lie group  $G$  on the Sobolev space  $\mathcal{H}_{\mathrm{inv}}^s (\mathcal{M})$ ,  $s > d / 2$ , with  $d = \dim (\mathcal{M} / G)$ . Assume that  $f^{\star}\in \mathcal{H}_{\mathrm{inv}}^{s\theta}(\mathcal{M})$  for some  $\theta \in (0,1]$ , and let  $s = \frac{d}{2} (\kappa +1)$  for a positive  $\kappa$ . Then,
$$
\mathbb {E} \left[ \mathcal {R} (\hat {f}) - \mathcal {R} (f ^ {\star}) \right] \leq 3 2 \left(\frac {1}{\kappa \theta} \frac {\omega_ {d}}{(2 \pi) ^ {d}} \frac {\sigma^ {2} \operatorname {v o l} (\mathcal {M} / G)}{n}\right) ^ {\theta s / (\theta s + d / 2)} \| f ^ {\star} \| _ {\mathcal {H} _ {\mathrm {i n v}} ^ {s \theta} (\mathcal {M})} ^ {d / (\theta s + d / 2)}, \tag {5}
$$
with the optimal regularization parameter
$$
\eta = \left(\frac {1}{2 \kappa \theta \| f ^ {\star} \| _ {\mathcal {H} _ {\mathrm {i n v}} ^ {s \theta} (\mathcal {M})} ^ {2}} \frac {\omega_ {d}}{(2 \pi) ^ {d}} \frac {\sigma^ {2} \operatorname {v o l} (\mathcal {M} / G)}{n}\right) ^ {\theta s / (\theta s + d / 2)}, \tag {6}
$$
where  $\omega_{d}$  is the volume of the unit ball in  $\mathbb{R}^d$ .
Theorem 4.1 allows to estimate the gain in sample complexity from making the hypothesis class invariant. Setting  $G = \{\mathrm{id}_G\}$  (i.e., the trivial group) recovers the standard generalization bound without group invariances. In particular, without invariances, the dimension  $d$  becomes  $\dim(\mathcal{M})$ , and the volume  $\operatorname{vol}(\mathcal{M}/G)$  becomes  $\operatorname{vol}(\mathcal{M})$ . Hence, group invariance can lead to a two-fold gain:
- Exponent: the dimension  $d$  in the exponent can be much smaller than the corresponding  $\dim (\mathcal{M})$ .
This definition is stronger from being shift-invariant; a kernel  $K$  is shift-invariant (with respect to the group  $G$ ) if and only if  $K(x,y) = K(\tau(x), \tau(y))$ , for all  $x, y \in \mathcal{M}$ , and all  $\tau$ . For any given shift-invariant kernel  $K$ , one can construct its corresponding  $G$ -invariant kernel  $\tilde{K}(x,y) = \int_{G} K(\tau(x), y)d\mu_G(\tau)$ , where  $\mu_G$  is the left-invariant Haar (uniform) measure of  $G$ .
- Effective number of samples: the number of samples is multiplied by
$$
\frac {\omega_ {\dim (\mathcal {M})} / (2 \pi) ^ {\dim (\mathcal {M})}}{\omega_ {d} / (2 \pi) ^ {d}} \cdot \frac {\operatorname {v o l} (\mathcal {M})}{\operatorname {v o l} (\mathcal {M} / G)}. \tag {7}
$$
The quantity (7) reduces to  $|G|$  if  $G$  is a finite group that efficiently acts on  $\mathcal{M}$  (i.e., if any group element acts non-trivially on the manifold). Intuitively, any sample conveys the information of  $|G|$  data points, which can be interpreted as having effectively  $n \times |G|$  samples (compared to non-invariant KRR with  $n$  samples). For groups of positive dimension, it measures how the group is contracting the volume of the manifold. Note that for finite groups, one always has  $\frac{\operatorname{vol}(\mathcal{M})}{\operatorname{vol}(\mathcal{M} / G)} \geq 1$ .
Dimension and volume for quotient spaces. In Theorem 4.1, the quotient space  $\mathcal{M} / G$  is defined as the set of all orbits  $[x] := \{\tau(x) : \tau \in G\}$ ,  $x \in \mathcal{M}$ , but  $\mathcal{M} / G$  is not always a (boundaryless) manifold (Appendix A.5 and A.6). Thus, it is not immediately possible to define its dimension and volume. The quotient space is a finite disjoint union of manifolds, each with its specific dimension/volume. In Appendix A.5 and A.6, we review the theory of quotients of manifolds, and observe that there exists an open dense subset  $\mathcal{M}_0 \subseteq \mathcal{M}$  such that  $\mathcal{M}_0 / G$  is open and dense in  $\mathcal{M} / G$ , and more importantly, it is a connected precompact manifold.  $\mathcal{M}_0 / G$  is called the principal part of the quotient space. It has the largest dimension among all the manifolds that make up the quotient space.
The projection map  $\pi : \mathcal{M}_0 \to \mathcal{M}_0 / G$  induces a metric on  $\mathcal{M}_0 / G$  and this allows us to define  $\operatorname{vol}(\mathcal{M} / G) \coloneqq \operatorname{vol}(\mathcal{M}_0 / G)$ . Note that  $\operatorname{vol}(\mathcal{M} / G)$  depends on the Riemannian metric, which itself might depend on the group  $G$  if we start from a base metric and then deform it to make the action isometric. The volume  $\operatorname{vol}(\mathcal{M}_0 / G)$  is computed with respect to the dimension of  $\mathcal{M}_0 / G$ , thus being nonzero even if  $\dim(\mathcal{M}_0 / G) < \dim(\mathcal{M})$ .
The effective dimension of the quotient space is defined as  $d \coloneqq \dim(\mathcal{M}_0 / G)$ . Alternatively, one can define the effective dimension as
$$
d := \dim (\mathcal {M}) - \dim (G) + \min  _ {x \in \mathcal {M}} \dim \left(G _ {x}\right), \tag {8}
$$
where  $G_{x} \coloneqq \{\tau \in G : \tau(x) = x\}$  is called the isotropic group of the action at point  $x \in \mathcal{M}$ . For example, if there exists a point  $x \in \mathcal{M}$  with the trivial isotropy group  $G_{x} = \{\mathrm{id}_{G}\}$ , then  $d = \dim(\mathcal{M}) - \dim(G)$ .
Remark 4.2. The invariant Sobolev space satisfies  $\mathcal{H}_{\mathrm{inv}}^s (\mathcal{M})\subseteq \mathcal{H}_{\mathrm{inv}}^{s\theta}(\mathcal{M})\subseteq L_{\mathrm{inv}}^2 (\mathcal{M})$ . If the regression function  $f^{\star}$  does not belong to the Sobolev space  $\mathcal{H}_{\mathrm{inv}}^s (\mathcal{M})$  (i.e.,  $\theta \in (0,1)$ ), the achieved exponent only depends on  $\theta s$  (i.e., the smoothness of the regression function  $f^{\star}$  and not the smoothness of the kernel). The bound decreases monotonically as  $s$  increases: smoother functions are easier to learn.
The next theorem states our minimax optimality result. For simplicity, we assume  $\theta = 1$ .
Theorem 4.3 (Minimax optimality). For any estimator  $\hat{f}$
$$
\sup  _ {\substack {f ^ {\star} \in \mathcal {H} _ {\mathrm {i n v}} ^ {s} (\mathcal {M}) \\ \| f ^ {\star} \| _ {\mathcal {H} _ {\mathrm {i n v}} ^ {s} (\mathcal {M})} = 1}} \mathbb {E} \left[ \mathcal {R} (\hat {f}) - \mathcal {R} (f ^ {\star}) \right] \geq C _ {\kappa} \left(\frac {\omega_ {d}}{(2 \pi) ^ {d}} \frac {\sigma^ {2} \operatorname{vol} (\mathcal {M} / G)}{n}\right) ^ {s / (s + d / 2)}, \tag{9}
$$
where  $C_{\kappa}$  is a constant only depending on  $\kappa$ , and  $\omega_{d}$  is the volume of the unit ball in  $\mathbb{R}^d$ .
An explicit formula for  $C_{\kappa}$  is given in the appendix. Note that the above minimax lower bound not only proves that the achieved bound by the KRR estimator is optimal, but also shows the optimality of the prefactor characterized in Theorem 4.1 with respect to the effective dimension  $d$  (up to multiplicative constants depending on  $\kappa$ ).
4.1 Proof Idea and Dimension Counting Bounds
To prove Theorem 4.1, we develop a general formula for the Fourier series of invariant functions on a manifold. In particular, we argue that a smooth  $G$ -invariant function  $f: \mathcal{M} \to \mathbb{R}$  corresponds to a smooth function  $\tilde{f}: \mathcal{M}/G \to \mathbb{R}$  on the quotient space  $\mathcal{M}/G$ , where  $\tilde{f}([x]) = f(x)$  for all  $x \in \mathcal{M}$ . Hence, we view the space of invariant functions as smooth functions on the quotient space.
The generalization bound essentially depends on a notion of dimension or complexity for this space, which allows bounding the bias and variance terms. We obtain this by controlling the eigenvalues of the Laplace-Beltrami operator, which is specifically suitable for Sobolev kernels.