SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The Laplace-Beltrami operator  $\Delta_g$  is the generalization of the usual definition of the Laplacian operator on the Euclidean space  $\mathbb{R}^d$  to any smooth manifold. It can be diagonalized in  $L^2(\mathcal{M})$  [12]. In particular, there exists an orthonormal basis  $\{\phi_\ell(x)\}_{\ell=0}^\infty$  of  $L^2(\mathcal{M})$  starting from the constant function  $\phi_0 \equiv 1$  such that  $\Delta_g\phi_\ell + \lambda_\ell\phi_\ell = 0$ , for the discrete spectrum  $0 = \lambda_0 < \lambda_1 \leq \lambda_2 \leq \dots$ . Let us call  $\{\phi_\ell(x)\}_{\ell=0}^\infty$  the Laplace-Beltrami basis for  $L^2(\mathcal{M})$ . Our notion of dimension of the function space is  $N(\lambda) \coloneqq \# \{\ell : \lambda_\ell \leq \lambda\}$ . It can be shown that if a Lie group  $G$  acts smoothly on the compact manifold  $\mathcal{M}$ , then  $G$  also acts on eigenspaces of the Laplace-Beltrami operator. Accordingly, we define the dimension  $N(\lambda; G)$  as the dimension of projecting the eigenspaces of the Laplace-Beltrami operator on  $\mathcal{M}$  onto the space of  $G$ -invariant functions, that is, the number of invariant eigenfunctions with eigenvalue up to  $\lambda$  (Appendix A.7).
Characterizing the asymptotic behavior of  $N(\lambda; G)$  is essential for proving our main results on the gain of invariances. Intuitively, the quantity  $N(\lambda; G) / N(\lambda)$  corresponds to the fraction of functions that are  $G$ -invariant. One of this paper's main contributions is to determine the exact asymptotic behavior of this quantity for the analysis of KRR. The tight bound on  $N(\lambda; G)$  can be of potential independent interest to other problems related to learning with invariances.
Theorem 4.4 (Dimension counting theorem). Let  $(\mathcal{M},g)$  be a smooth connected compact boundaryless Riemannian manifold of dimension  $\dim (\mathcal{M})$  and let  $G$  be a compact Lie group of dimension  $\dim (G)$  acting isometrically on  $(\mathcal{M},g)$ . Recall the definition of the effective dimension of the quotient space  $d\coloneqq \dim (\mathcal{M}) - \dim (G) + \min_{x\in \mathcal{M}}\dim (G_x)$ . Then,
$$
N (\lambda ; G) = \frac {\omega_ {d}}{(2 \pi) ^ {d}} \operatorname {v o l} (\mathcal {M} / G) \lambda^ {d / 2} + \mathcal {O} \left(\lambda^ {\frac {d - 1}{2}}\right), \tag {10}
$$
as  $\lambda \to \infty$ , where  $\omega_{d}$  is the volume of the unit ball in  $\mathbb{R}^d$ .
In Appendix B, we prove a generalized version of the above bound (i.e., a local version).
To see how this counting bound relates to Sobolev spaces, note that by Mercer's theorem, a positive-definite symmetric (PDS) kernel  $K:\mathcal{M}\times \mathcal{M}\to \mathbb{R}$  can be diagonalized in an appropriate orthonormal basis of functions in  $L^2 (\mathcal{M})$ . Indeed, the kernel of the Sobolev space  $\mathcal{H}^s (\mathcal{M})\subseteq L^2 (\mathcal{M})$  is diagonalizable in the Laplace-Beltrami basis:
$$
K _ {\mathcal {H} ^ {s} (\mathcal {M})} (x, y) = \sum_ {\ell = 0} ^ {\infty} \min  \left(1, \lambda_ {\ell} ^ {- s}\right) \phi_ {\ell} (x) \phi_ {\ell} (y), \tag {11}
$$
where  $\phi_{\ell},\ell = 0,1,\ldots$  , form a basis for  $L^2 (\mathcal{M})$  such that  $\Delta_g\phi_\ell +\lambda_\ell \phi_\ell = 0$  for each  $\ell$  . For the  $G$  -invariant RKHS  $\mathcal{H}_{\mathrm{inv}}^s (\mathcal{M})$  , the sum is restricted to  $G$  -invariant eigenfunctions (Appendix A.9). It is evident that Theorem 4.4 provides an important tool for analyzing the Sobolev kernel development with respect to the eigenfunctions of Laplacian.
4.2 Proof Idea of the Dimension Counting Theorem
For Riemannian manifolds, Weyl's law (Appendix A.4) determines the asymptotic distribution of eigenvalues. The bound in Theorem 4.4 indeed corresponds to Weyl's law, if we write it in terms of the quotient space  $\mathcal{M} / G$ . But, in general, Weyl's law does not apply to the quotient space  $\mathcal{M} / G$ . So this intuition is not rigorous. First, the quotient space is not always a manifold (Appendix A.5). Second, even if we restrict our attention to the principal part  $\mathcal{M}_0 / G$  discussed above, which is provably a manifold, other complications arise.
In particular, the quotient  $\mathcal{M}_0 / G$  can exhibit a boundary, even if the original space is boundaryless. For a concrete example, consider the circle  $\mathbb{S}^1 = \{(x,y):x^2 +y^2 = 1\}$ , parameterized by the angle  $\theta \in [0,2\pi)$ , under the isometric action  $G = \{\mathrm{id}_G,\tau \}$ , where  $\tau (\theta) = \pi -\theta$  is the reflection across the  $y$ -axis. The resulting space is a semicircle with two boundary points  $(0,\pm 1)$ .
4Many kernels in practice satisfy this condition, e.g., any dot-product kernel on a sphere. While we present the results of this paper for Sobolev kernels, one can use any kernel satisfying the condition in Proposition A.9 and apply the same techniques to obtain its convergence rates.
If the space has a boundary, then a dimension counting result like Weyl's law for manifolds is only true with an appropriate boundary condition for finding the eigenfunctions. In general, different boundary conditions can lead to completely different function spaces for manifolds with boundaries. In the proof, we show that the projections of invariant functions onto the quotient space satisfy the Neumann boundary condition on the (potential) boundaries of the quotient space, thereby exactly characterizing the space of invariant functions, which can indeed be of independent interest.
To see how the Neumann boundary condition appears, consider the circle again and note that its eigenfunctions are the constant function  $\phi_0 \equiv 1$ ,  $\sin(k\theta)$ , and  $\cos(k\theta)$ , with eigenvalues  $\lambda = k^2$ ,  $k \in \mathbb{N}$ . Observe that every eigenvalue is of multiplicity two, except for the zero eigenvalue, which has a multiplicity of one. For the quotient space  $\mathbb{S}^1 / G$ , however, the eigenfunctions are just the constant function,  $\sin((2k + 1)\theta)$ , and  $\cos(2k\theta)$ ,  $k \in \mathbb{Z}$  (note how roughly half of the eigenfunctions survived, as  $|G| = 2$ ). In particular, the boundary points  $(0, \pm 1)$  satisfy the Neumann boundary condition, while the Dirichlet boundary condition fails to hold; look at the eigenfunctions at  $\theta = \pi/2$ . More precisely, if we consider the Dirichlet boundary condition, then we get a function space that includes only functions vanishing at the boundary points:  $\phi(\pi/2) = \phi(3\pi/2) = 0$ . This clearly does not correspond to the space of invariant functions. We generalize this idea in our proof to any manifold and group using differential geometric tools (see Appendix A.5 and Appendix A.6).
In the above example, the boundary points come from the fact that the group action has non-trivial fixed points, i.e.,  $(0, \pm 1)$  are the fixed points. If the action is free, meaning that we have only trivial fixed points, then the quotient space is indeed a boundaryless manifold. Thus, the challenges towards the proof arise from the existence of non-trivial fixed points.
Comparison to prior work. Lastly, we discuss an example on the two-dimensional flat torus  $\mathbb{T}^2 = \mathbb{R}^2 / 2\pi \mathbb{Z}^2$ , which shows that the proof ideas in [5] are indeed not applicable for general manifolds even with finite group actions. For this boundaryless manifold, consider the isometric action  $G = \{\mathrm{id}_G, \tau\}$ , where  $\tau(\theta_1, \theta_2) = (\theta_1 + \pi, \theta_2)$ , and note that the quotient space  $\mathbb{T}^2 / G$  is again a torus. In this case, the eigenfunctions of  $\mathbb{T}^2$  are the functions  $\exp(ik_1x + ik_2y)$ , for all  $k_1, k_2 \in \mathbb{Z}$ , with eigenvalue  $\lambda = k_1^2 + k_2^2$ . But eigenfunctions of the quotient space are those with even  $k_1$ . This means that some eigenspaces of  $\mathbb{T}^2$  (such as those with eigenvalue  $\lambda = 2(2n + 1)^2$ ) are completely lost after projection onto the space of invariant functions. This means that the method used in [5] fails to give a non-trivial bound here, because it relies on the fraction of eigenfunctions that survive in each eigenspace. Note that in this example, the quotient space is boundaryless, and the action is free.
4.3 Application to Finite-Dimensional Kernels
Applications of Theorem 4.4 are not limited to Sobolev spaces. As an example, we study KRR with finite-dimensional PDS kernels  $K:\mathcal{M}\times \mathcal{M}\to \mathbb{R}$  with an RKHS  $\mathcal{H}\subseteq L^2 (\mathcal{M})$  under invariances (the inclusion must be understood in terms of Hilbert spaces, i.e., the inner product on  $\mathcal{H}$  is just the usual inner product defined on  $L^2 (\mathcal{M})$ , making it completely different from Sobolev spaces). Examples of these finite-dimensional spaces are random feature models and two-layer neural networks in the lazy training regime. In this section, we will relate the generalization error of KRR to the average amount of fluctuations of functions in the space and use our dimension counting result to study the gain of invariances.
To formalize this notion of complexity, we need to review some definitions. We measure fluctuation via the Dirichlet form  $\mathcal{E}$ , a bilinear form defined as
$$
\mathcal {E} \left(f _ {1}, f _ {2}\right) := \int_ {\mathcal {M}} \left\langle \nabla_ {g} f _ {1} (x), \nabla_ {g} f _ {2} (x) \right\rangle_ {g} d \operatorname {v o l} _ {g} (x), \tag {12}
$$
for any two smooth functions  $f_1, f_2: \mathcal{M} \to \mathbb{R}$ . It can be easily extended (continuously) to any Sobolev space  $\mathcal{H}^s(\mathcal{M})$ ,  $s \geq 1$ . For each  $f \in \mathcal{H}^s(\mathcal{M})$ , the diagonal quantity  $\mathcal{E}(f, f)$  is called the Dirichlet energy of the function, and is a measure of complexity of a function. Functions with low Dirichlet energy have little fluctuation; intuitively, those have low (normalized) Lipschitz constants on average. Since  $\mathcal{E}(af, af) = |a|^2 \mathcal{E}(f, f)$ , it is more accurate to restrict to the case  $\| f \|_{L^2(\mathcal{M})} = 1$  while studying low-energy functions.
One important example of a finite-dimensional function space is the space of  $G$ -invariant low-energy functions, which is generated by a finite-dimensional  $G$ -invariant kernel  $K$ :
$$
K (x, y) = \sum_ {\ell = 0} ^ {D - 1} \phi_ {\ell} (x) \phi_ {\ell} (y), \tag {13}
$$
for any  $x, y \in \mathcal{M}$ . The non-zero  $G$ -invariant eigenfunctions  $\phi_{\ell}$  are sorted with respect to their eigenvalues (Appendix A.9). Clearly,  $K$  is a kernel of dimension  $D$ , and it is diagonal in the basis of the Laplace-Beltrami operator's eigenfunctions. The RKHS of  $K$  is also of finite dimension  $\dim(\mathcal{H}_G) = D$  and can be written as
$$
\mathcal {H} _ {G} := \left\{f \in L ^ {2} (\mathcal {M}): f = \sum_ {\ell = 0} ^ {D - 1} \langle f, \phi_ {\ell} \rangle_ {L ^ {2} (\mathcal {M})} \phi_ {\ell} \right\} \subseteq L _ {\text {i n v}} ^ {2} (\mathcal {M}, G). \tag {14}
$$
To obtain generalization bounds, we will need to bound a complexity measure of the space of low-energy invariant functions. As a complexity measure for finite function spaces  $\mathcal{H} \subseteq L_{\mathrm{inv}}^2(\mathcal{M}, G)$ , we use the Dirichlet energy  $\mathcal{E}(f, f)$ :
$$
\mathcal {L} (\mathcal {H}) := \max  _ {f \in \mathcal {H}} \left\{\mathcal {E} (f, f): \| f \| _ {L ^ {2} (\mathcal {M})} \leq 1 \right\}. \tag {15}
$$
For a vector space  $\mathcal{H}$ , larger  $\mathcal{L}(\mathcal{H})$  corresponds to having functions with more (normalized) fluctuation. Thus,  $\mathcal{L}(V)$  is a notion of complexity for the vector space  $V$ . The following proposition shows how  $\mathcal{H}_G$  is the simplest subspace of  $L^2 (\mathcal{M})$  with dimension  $D$ .
Proposition 4.5. For any  $D$ -dimensional vector space  $\mathcal{H} \subseteq L_{\mathrm{inv}}^2(\mathcal{M}, G)$ ,
$$
\mathcal {L} (\mathcal {H}) \geq \lambda_ {D - 1}, \tag {16}
$$
and the equality is only achieved when  $\mathcal{H} = \mathcal{H}_G$  with  $D = \dim (\mathcal{H}_G)$ . In particular, if  $\mathcal{L}(\mathcal{H}) < \infty$ , then  $\mathcal{H}$  is of finite dimension. The eigenvalues are sorted according to  $G$ -invariant eigenspaces (Appendix A.9).
Using the dimension counting bound in Theorem 4.4, we can explicitly relate the dimension of  $\mathcal{H}_G$  to its complexity  $\mathcal{L}(\mathcal{H}_G)$ . This will be useful for studying the gain of invariances for finite-dimensional kernels.
Theorem 4.6 (Dimension of the space of low-energy functions). Under the assumptions in Theorem 4.4, one has the following relation between the dimension of the vector space  $\mathcal{H}_G$  and its complexity  $\mathcal{L}(\mathcal{H}_G)$ :
$$
\dim (\mathcal {H} _ {G}) = \frac {\omega_ {d}}{(2 \pi) ^ {d}} \operatorname {v o l} (\mathcal {M} / G) \mathcal {L} \left(\mathcal {H} _ {G}\right) ^ {d / 2} + \mathcal {O} \left(\mathcal {L} \left(\mathcal {H} _ {G}\right) ^ {\frac {d - 1}{2}}\right), \tag {17}
$$
where  $\omega_{d}$  is the volume of the unit ball in  $\mathbb{R}^d$
Given the above result, in conjunction with Proposition 4.5, we can obtain the following generalization error for any finite-dimensional RKHS  $\mathcal{H} \subseteq L^2(\mathcal{M})$ .
Corollary 4.7 (Convergence rate of KRR with invariances for finite dimensional kernels). Under the assumptions in Theorem 4.4, for KRR with an arbitrary finite-dimensional  $G$ -invariant RKHS  $\mathcal{H} \subseteq L^2(\mathcal{M})$ ,
$$
\mathbb {E} \left[ \mathcal {R} (\hat {f}) - \mathcal {R} (f ^ {\star}) \right] \lesssim \left(\frac {\omega_ {d}}{(2 \pi) ^ {d}} \frac {\sigma^ {2} \operatorname {v o l} (\mathcal {M} / G)}{n}\right) \mathcal {L} (\mathcal {H}) ^ {d / 2} \| f ^ {\star} \| _ {L ^ {2} (\mathcal {M})} ^ {2}, \tag {18}
$$
where  $\lesssim$  hides absolute constants. Moreover, the upper bound is minimax optimal if  $\mathcal{H} = \mathcal{H}_G$  (similar to Theorem 4.3).
Corollary 4.7 shows the same gain of invariances in terms of sample complexity as Theorem 4.1. Note that in the asymptotic analysis for the above corollary, we assume that  $\mathcal{L}(\mathcal{H})$  is large enough, allowing us to use Theorem 4.6.
5 Examples and Applications
Next, we make our general results concrete for a number of popular learning settings with invariances. This yields results for kernel versions of popular corresponding architectures.
5.1 Sets
When learning with sets, each data instance is a subset  $\{x_{1}, x_{2}, \ldots, x_{m}\}$  of elements  $x_{i} \in \mathcal{X}$ ,  $i \in [m]$ , from a given space  $\mathcal{X}$ . A set is invariant under permutations of its elements, i.e.,
$$
\left\{x _ {1}, x _ {2} \dots , x _ {m} \right\} = \left\{x _ {\sigma_ {1}}, x _ {\sigma_ {2}} \dots , x _ {\sigma_ {m}} \right\}, \tag {19}
$$
where  $\sigma :[m]\to [m]$  can be any permutation. A successful permutation invariant architecture for learning on sets is DeepSets [70]. Similarly, PointNets are a permutation invariant architecture for point clouds [53, 54]. To analyze learning with sets and kernel versions of these architectures using our formulation, we assume sets of fixed cardinality  $m$ . If the space  $\mathcal{X}$  has a manifold structure, then one can identify each data instance as a point on the product manifold
$$
\mathcal {M} = \mathcal {X} ^ {m} = \underbrace {\mathcal {X} \times \mathcal {X} \cdots \times \mathcal {X}} _ {m}. \tag {20}
$$
The task is invariant to the action of the symmetric group  $S_{m}$  on  $\mathcal{M}$ ; each  $\sigma \in S_{m}$  acts on  $\mathcal{M}$  by permuting the coordinates as in Equation (19). This action is isometric, and we have  $\dim(S_{m}) = 0$  and  $|S_{m}| = 1 / m!$ . Theorem 4.1 hence implies that the sample complexity gain from permutation invariance is having effectively  $n \times m!$  samples, where  $n$  is the number of observed sets. In fact, this result holds (for KRR) for any space  $\mathcal{X}$  with a manifold structure.
5.2 Images
For images, we need translation invariant models. For instance, Convolutional Neural Networks (CNNs) [31, 28] compute translation invariant image representations. Each image is a 2D array  $(x_{i,j})_{i,j=0}^{m-1}$  such that  $x_{i,j} \in \mathcal{X}$  for a space  $\mathcal{X}$  (e.g., for RGB,  $\mathcal{X} \subseteq \mathbb{R}^3$  is a compact subset). If  $\mathcal{X}$  has a manifold structure, then one can identify each image with a point on the manifold
$$
\mathcal {M} = \bigotimes_ {i, j = 0} ^ {m - 1} \mathcal {X} ^ {i, j}, \tag {21}
$$
where  $\mathcal{X}^{i,j}$  is a copy of  $\mathcal{X}$ . The learning task is invariant under the action of the finite group  $(\mathbb{Z}/m\mathbb{Z}) \times (\mathbb{Z}/m\mathbb{Z})$  on  $\mathcal{M}$  by shifting pixels: each  $(p,q) \in (\mathbb{Z}/m\mathbb{Z}) \times (\mathbb{Z}/m\mathbb{Z})$  corresponds to the isometry  $(x_{i,j})_{i,j=1}^{m} \mapsto (x_{i+p,j+q})_{i,j=1}^{m}$  (the sum is understood modulo  $m$ ). As a result, the sample complexity gain corresponds to having effectively  $n \times m^2$  samples, where  $n$  is the number of images.
5.3 Point Clouds
3D point clouds have rotation, translation, and permutation symmetries. Tensor field neural networks [62] respect these invariances. We view each 3D point cloud as a set  $\{x_1, x_2, \ldots, x_m\}$  such that  $x_i \in \mathbb{R}^3 / \mathbb{Z}^3 \equiv [0,1]^3$ , which is essentially a point on the manifold  $\mathcal{M} = (\mathbb{R}^3 / \mathbb{Z}^3)^m$  with  $\dim(\mathcal{M}) = 3m$ . The learning task is invariant with respect to permuting the coordinates of  $\mathcal{M}$ , translating all points  $x_i \mapsto x_i + r$  for some  $r \in \mathbb{R}^3$ , and jointly rotating all points,  $x_i \mapsto Qx_i$  for an orthogonal matrix  $Q$ . We denote the group defined by those three operations as  $G$  and observe that  $\dim(G) = 6$ . Thus, the gains of invariances in sample complexity are (1) reducing the dimension  $d$  of the space from  $3m$  to  $3m - 6$ , and (2) having effectively  $n \times m!$  samples, where  $n$  is the number of point clouds.
5.4 Sign Flips of Eigenvectors
SignNet [37] is a recent architecture for learning functions of eigenvectors in a spectral decomposition. Each data instance is a sequence of eigenvectors  $(v_{1}, v_{2}, \ldots, v_{m})$ ,  $v_{i} \in \mathbb{R}^{d}$ , and flipping the sign of an eigenvector  $v_{i} \rightarrow -v_{i}$  does not change its eigenspace. The spectral data can be considered as a point on the manifold  $\mathcal{M} = (\mathbb{S}^{d-1})^{m}$  (where  $\mathbb{S}^{d-1}$  is the  $(d-1)$ -dimensional sphere), while the task is invariant to all  $2^{m}$  possibilities of sign flips. The sample complexity gain of invariances is thus having effectively  $n \times 2^{m}$  samples, where  $n$  is the number of spectral data points.
5.5 Changes of Basis for Eigenvectors
BasisNet [37] represents spectral data with eigenvalue multiplicities. Each input instance is a sequence of eigenspaces  $(V_{1}, V_{2}, \ldots, V_{p})$ , and each  $V_{i}$  is represented by an orthonormal basis such
as  $(v_{i,1}, v_{i,2}, \ldots, v_{i,m_i}) \in (\mathbb{R}^d)^{m_i}$ . This is the Stiefel manifold with dimension  $dm_i - \frac{m_i(m_i + 1)}{2}$ . Thus, the spectral data lie on a manifold of dimension
$$
\dim (\mathcal {M}) = \sum_ {i = 1} ^ {p} \left(d m _ {i} - m _ {i} \left(m _ {i} + 1\right) / 2\right). \tag {22}
$$
The vector spaces' representations are invariant to a change of basis, i.e., the group action is defined as  $(v_{i,1}, v_{i,2}, \ldots, v_{i,m_i}) \mapsto (Qv_{i,1}, Qv_{i,2}, \ldots, Qv_{i,m_i})$  for any orthogonal matrix  $Q$  that fixes the eigenspace  $V_i$ . If  $G$  denotes this group of invariances, then
$$
\dim (G) = \sum_ {i = 1} ^ {p} \left(m _ {i} \left(m _ {i} - 1\right) / 2\right). \tag {23}
$$
Thus, the gain of invariances is a reduction of the manifold's dimension to  $\sum_{i=1}^{p}(dm_i - m_i^2)$ . For example, if  $m_i = m$  for all  $i$ , then with  $d = pm$  we get  $\dim(\mathcal{M}) = d^2 - \frac{1}{2}d(m + 1)$  while after the reduction we have  $\dim(\mathcal{M}/G) = d^2 - dm$ . In this example, the quotient space is the Grassmannian manifold.
5.6 Learning on Graphs
Each weighted, possibly directed graph on  $m$  vertices with initial node attributes can be naturally encoded by its adjacency matrix  $A \in \mathbb{R}^{m \times m}$ . Thus, the space of all weighted directed graphs on  $m$  vertices corresponds to a compact manifold  $\mathcal{M} \subseteq \mathbb{R}^{m \times m}$  if the weights are restricted to a bounded set. Learning tasks on graphs are invariant to permutations of rows and columns, i.e., the action of the symmetric group as  $A \mapsto P^{-1}AP$  for any permutation matrix  $P$ . For instance, Graph Neural Networks (GNNs) and graph kernels [64] implement this invariance. The sample complexity gain from invariances is thus evident; it corresponds to having effectively  $n \times m!$  samples, where  $n$  is the number of sampled graphs.
5.7 Hyperbolic Spaces, Tori, etc.
One important feature of the results in this paper is that they are not restricted to compact submanifolds of Euclidean spaces. In particular, the results are also valid for compact hyperbolic spaces; see [51] for a survey on applications of hyperbolic spaces in machine learning. Another type of space where our results are still valid are tori, i.e.,  $\mathbb{T}^d \coloneqq (\mathbb{S}^1)^d$ . Tori naturally occur for modeling joints in robot control [38]. In fact, Riemannian manifolds are beneficial in a broader context for learning in robotics, e.g., arising from constraints, as surveyed in [10]. Our results apply to invariant learning in all of these settings, too.
In this work, we derived new generalization bounds for learning with invariances. These generalization bounds show a two-fold gain in sample complexity: (1) in the dimension term in the exponent, and (2) in the effective number of samples. Our results significantly generalize the range of settings where the bounds apply. In particular, (1) goes beyond prior work, since it applies to groups of positive dimension, whereas prior work assumed finite dimensions. At the heart of our analysis is a new dimension counting bound for invariant functions on manifolds, which we expect to be useful more generally for analyzing learning with invariances. We prove this bound via differential geometry and show how to overcome several technical challenges related to the properties of the quotient space.
The authors extend their appreciation to Semyon Dyatlov and Henrik Christensen for their valuable recommendations and inspiration. This work was supported by Office of Naval Research grant N00014-20-1-2023 (MURI ML-SCOPE), NSF award CCF-2112665 (TILOS AI Institute), NSF award 2134108, and NSF TRIPODS program (award DMS-2022448).