On the Sample Complexity of Learning under Geometric Stability

1 Introduction . Learning from high-dimensional data is known to be statistically intractable without strong assumptions on the problem . A canonical example is learning Lipschitz functions , which generally requires a number of samples exponential in the dimension due to the curse of dimensionality ( e.g. , [ 31 ] ) . Many high-dimensional machine learning problems involve highly structured data such as images , text , or graphs , and may exhibit invariance to certain transformations of the input data , such as permutations , translations or rotations , and near invariance to small deformations . More precisely , if X is the high-dimensional data domain , and G is a set of transformations σ : X → X , the learning task can be alleviated if one knows in advance that the target function f varies smoothly under transformations in G : |f ( σ · x ) − f ( x ) | is uniformly small over x ∈ X for σ ∈ G. To further motivate this property , it is useful to view the data domain X as a space of signals X = L2 ( Ω ; R ) defined over a geometric domain Ω , such as a 2d grid . The set of transformations G can then be articulated in terms of Ω rather than X , a much simpler geometric object , and then lifted into X by composition : if σ : Ω → Ω , and x ∈ X then ( σ · x ) ( u ) : = x ( σ−1 ( u ) ) for every u ∈ Ω . The smoothness to transformations can thus be interpreted as a form of geometric stability . In this paper , we quantify the sample complexity gains brought by geometric stability . Concretely , we consider target functions f defined on the sphere X = Sd−1 in d dimensions with finite L2 ( Sd−1 ) norm . In this case , we view the geometric domain as the discrete 1d grid Ω = [ 1 , . . . , d ] , and consider geometric transformations G as subsets of the symmetric group of permutations of d ∗Center for Data Science , New York University . †Courant Institute for Mathematical Sciences , New York University . ‡Center for Data Science and Courant Institute for Mathematical Sciences , New York University . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . elements . Given a set G ( not necessarily a group ) , we consider the smoothing operator given by SGf ( x ) = 1 |G| ∑ σ∈G f ( σ · x ) for f ∈ L 2 ( Sd−1 ) , and assume that our target function f is geometrically stable , in the sense that f = SGg for some g ∈ L 2 ( Sd−1 ) . In words , the smoothing operator SG replaces the prediction f ( x ) by the average over transformations of x . In particular , functions which are invariant under the action of σ ∈ G , namely f ( σ · x ) = f ( x ) , σ ∈ G , x ∈ Sd−1 , ( 1 ) are also stable , with f = SGf . Building on the recent work [ 23 ] , we proceed by studying harmonic decompositions of such functions using spherical harmonics [ 14 ] , which generalize Fourier series on the circle to higher dimensions . This allows us to obtain rates of approximation for invariant and geometrically stable functions with varying levels of smoothness , and to study the generalization properties of invariant kernel methods using kernels defined on the sphere . Specifically , our main contributions are : • By comparing spectral properties of usual kernels on the sphere with invariant ones , we find that the latter provide improvements in sample complexity by a factor of the order of the size of the group when the sample size is large enough ( Section 3 ) . • We study how this improvement factor varies with sample size , in terms of the structure of the group and on spectral properties of the permutation matrices it contains ( Section 4 ) . • We extend the invariance analysis to geometrically stable functions , establishing similar gains in sample complexity that depend on the size of the transformation subset ( Section 5 ) . Our proofs rely on comparing the dimension of invariant and non-invariant spherical harmonics at a given degree , and showing that their ratio decays to the inverse group size as the degree tends to infinity . In contrast to [ 23 ] , we consider the dimension to be fixed and study non-parametric rates of convergence for potentially non-smooth target functions and general groups of permutations , while they consider a different regime in high dimension and focus on invariance to translation groups . Related work . Invariance and deformation stability have been analysed using convolutional neural network-type architectures such as the scattering transform [ 22 , 7 ] , or convolutional kernels [ 6 , 20 ] . While these works characterise the stability in terms of the dyadic structure of convolutional filters ( such as wavelets ) , they do not cover a statistical analysis of sample complexity . Similarly , models of compositional functions such as those in [ 11 , 24 , 27 ] study the benefit of hierarchical representations with local connectivity for approximation , while [ 19 , 21 ] study benefits of local connectivity with optimization-based algorithms ; yet these works do not consider invariance or stability . [ 23 ] studies similar benefits of invariance but in a different , high-dimensional , regime where only polynomials can be learned , focusing on translation groups , while we consider arbitrary groups or sets of permutations in fixed dimension . [ 15 ] also sudies benefits of group invariance , but focuses on linear models , and only considers interpolating estimators . [ 30 ] study general generalization bounds of invariant classifiers that scale exponentially with the dimension , which would be pessimistic under our assumptions . [ 10 ] study benefits of equivariant kernels in structured prediction problems . [ 13 ] studies generalisation advantages of CNNs over fully-connected models , while our focus is on non-parametrics . 2 Preliminaries . In this section , we describe our setup and provide some background on harmonic decompositions on the sphere , and how these are affected by invariance . Statistical learning setup . We consider a supervised learning problem where the data distribution ρ on input-label pairs ( x , y ) is such that x ∈ Sd−1 and E [ y|x ] = f∗ ( x ) for some target function f∗in L2 ( Sd−1 ) . For simplicity , we will assume that x is uniformly distributed on the sphere , and denote the uniform measure on Sd−1 by dτ . We consider a regression setup with L2 risk given by R ( f ) = E ( x , y ) ∼ρ [ ( f ( x ) − y ) 2 ] . For a given estimator f̂n based on n samples from ρ , the goal is then to obtain generalization bounds as a function of n on the excess risk E [ R ( f̂n ) ] −R ( f ∗ ) = E [ ‖f̂n − f ∗‖2L2 ( dτ ) ] , ( 2 ) where the expectation is over the n samples . Such bounds are well-studied for various classes of target functions f∗ such as smoothness classes , and estimators such as kernel ridge regression . These are typically studied through harmonic decompositions of f∗ and of a kernel function in appropriate L2 bases , which then relate function regularity and decays of Fourier coefficients . Harmonic analysis on the sphere . When considering functions in L2 ( dτ ) , an appropriate choice of orthonormal basis is that of spherical harmonic polynomials [ 1 , 14 ] . More precisely , denote { Yk , j } N ( d , k ) j=1 denote an orthonormal basis of the space Vd , k of spherical harmonics of degree k , i.e. , homogeneous harmonic polynomials of degree k , where N ( d , k ) = 2k+d−2k ( k+d−3 d−2 ) . Then , the collection { Yk , j : k ≥ 0 , j = 1 , . . . , N ( d , k ) } forms an orthonormal basis of L 2 ( dτ ) , so that any function f ∈ L2 ( dτ ) may be written f ( x ) = ∑ k≥0 N ( d , k ) ∑ j=1 ak , jYk , j ( x ) , ( 3 ) with ∑ k ∑N ( d , k ) j=1 a 2 k , j < ∞ . Similarly , any dot-product kernel K ( x , x ′ ) = κ ( 〈x , x′〉 ) on the sphere may be written κ ( 〈x , x′〉 ) = ∑ k≥0 µk N ( d , k ) ∑ j=1 Yk , j ( x ) Yk , j ( x ′ ) , ( 4 ) where µk is given by µk = ωd−2 ωd−1 ∫ 1 −1 κ ( t ) Pd , k ( t ) ( 1−t 2 ) ( d−3 ) /2dt . Here , ωp−1 is the surface measure of the sphere in p dimensions , and Pd , k are Legendre or Gegenbauer polynomials of degree k in d dimensions ( normalized with Pd , k ( 1 ) = 1 ) , which form an orthogonal basis of L 2 ( [ −1 , 1 ] , dw ) , with dw ( t ) = ( 1− t2 ) ( d−3 ) /2dt . When the kernel K is positive definite and is used in the context of kernel ridge regression with data uniformly distributed on the sphere , then the µk also correspond to the eigenvalues of the covariance operator . These eigenvalues and their decay then control the statistical properties of the kernel ridge regression estimator [ 8 ] . Spherical harmonics and group-invariant functions . In order to describe harmonic decompositions of functions satisfying the group invariance property ( 1 ) for a discrete group G , we follow [ 23 ] and define the symmetrization operator SGf ( x ) = 1 |G| ∑ σ∈G f ( σ · x ) . ( 5 ) This operator acts as a projection from L2 ( dτ ) to a subset thereof which contains invariant functions . It can be shown that the spaces Vd , k of spherical harmonics of degree k are stable by SG [ 23 ] , and we may then define an orthonormal basis of V d , k : = SGVd , k consisting of invariant spherical harmonics { Y k , j } N ( d , k ) j=1 . We then have the following lemma . Lemma 1 ( Representation of projection [ 23 ] ) . For any k ≥ 0 , we have γd ( k ) : = N ( d , k ) N ( d , k ) = 1 |G| ∑ σ∈G Ex [ Pd , k ( 〈σ · x , x〉 ) ] . ( 6 ) The quantity γd ( k ) will play an important role in determining the gains in sample complexity brought by invariance . We will show in Section 4 that γd ( k ) converges to 1/|G| for large k , with an asymptotic behavior that is governed by spectral properties of the elements of the group .

Summary: In the context of kernel ridge regression, the authors study the improvements in generalization error for estimating target functions that are invariant or stable under a subset of symmetry transformations, when using kernels that have the same invariance or stability properties. Specifically, the authors study estimation of functions on the d-dimensional sphere, and consider symmetry transformations given by permutations of the d coordinates. The main results are generalization error upper bounds in two contexts: (1) When the subset of symmetry transformations form a subgroup of the symmetric group, the target function is exactly invariant under this subgroup, and the invariant kernel is given by a symmetrization of an inner-product kernel---Theorem 5. (2) When the subset of symmetry transformations can be arbitrary subset of permutations, and both the target function and kernel are given by an averaging operation applied to an initial function and inner-product kernel, with respect to some distribution over the transformations of this subset---Theorem 13.  The improvement in generalization error for the invariant kernel over the initial inner-product kernel comes from a reduction in the degrees of freedom associated to the ridge regressor, which may be upper-bounded by comparing the dimension of invariant spherical harmonics with that of all spherical harmonics at a given frequency k. In the analyzed setting where symmetry transformations are |G| permutations of d coordinates, the authors show that this ratio of dimensions converges to 1/|G| in the limit of large frequency k -> infty, which is the relevant limit for large sample sizes n -> infty. The authors quantify this rate of convergence in k for a number of different examples of symmetry subgroups.