Provably Strict Generalisation Benefit for Invariance in Kernel Methods

1 Introduction . Recently , there has been significant interest in models that are invariant to the action of a group on their inputs . It is believed that engineering models in this way improves sample efficiency and generalisation . Intuitively , if a task has an invariance , then a model that is constructed to be invariant ahead of time should require fewer examples to generalise than one that must learn to be invariant . Indeed , there are many application domains , such as fundamental physics or medical imaging , in which the invariance is known a priori [ 29 , 32 ] . Although this intuition is certainly not new ( e.g . [ 33 ] ) , it has inspired much recent work ( for instance , see [ 36 , 15 ] ) . However , while implementations and practical applications abound , until very recently a rigorous theoretical justification for invariance was missing . As pointed out in [ 8 ] , many prior works such as [ 28 , 24 ] provide only worst-case guarantees on the performance of invariant algorithms . It follows that these results do not rule out the possibility of modern training algorithms automatically favouring invariant models , irrespective of the choice of architecture . Steps towards a more concrete theory of the benefit of invariance have been taken by [ 8 , 20 ] and our work is a continuation along the path set by [ 8 ] . In this work we provide a precise characterisation of the generalisation benefit of invariance in kernel ridge regression . In contrast to [ 28 , 24 ] , this proves a provably strict generalisation benefit for invariant , feature-averaged models . In deriving this result , we provide insights into the structure of reproducing kernel Hilbert spaces in relation to invariant functions that we believe will be useful for analysing invariance in other kernel algorithms . The use of feature averaging to produce invariant predictors enjoys both theoretical and practical success [ 17 , 9 ] . For the purposes of this work , feature averaging is defined as training a model as normal ( according to any algorithm ) and then transforming the learned model to be invariant . This transformation is done by orbit-averaging , which means projecting the model on the space of invariant functions using the operator O introduced in Section 2.3 . Kernel methods have a long been a mainstay of machine learning ( see [ 30 , Section 4.7 ] for a brief historical overview ) . Kernels can be viewed as mapping the input data into a potentially infinite 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . dimensional feature space , which allows for analytically tractable inference with non-linear predictors . While modern machine learning practice is dominated by neural networks , kernels remain at the core of much of modern theory . The most notable instance of this is the theory surrounding the neural tangent kernel [ 11 ] , which states that the functions realised by an infinitely wide neural network belong to a reproducing kernel Hilbert space ( RKHS ) with a kernel determined by the network architecture . This relation has led to many results on the theory of optimisation and generalisation of wide neural networks ( e.g . [ 14 , 3 ] ) . In the same vein , via the NTK , we believe the results of this paper can be extended to study wide , invariant neural networks . 1.1 Summary of Contributions . This paper builds towards a precise characterisation of the benefit of incorporating invariance in kernel ridge regression by feature averaging . Lemma 3 , given in Section 3 , forms the basis of our work , showing that the action of the group G on the input space induces an orthogonal decomposition of the RKHS H as H = H⊕H⊥ where each term is an RKHS and H consists of all of the invariant functions in H. We stress that , while the main results of this paper concern kernel ridge regression , Lemma 3 holds regardless of training algorithm and could be used to explore invariance in other kernel methods . Our main results are given in Section 4 and we outline them here . We define the generalisation gap ∆ ( f , f ′ ) for two predictors f , f ′ as the difference in their test errors . If ∆ ( f , f ′ ) > 0 then f ′ has strictly better test performance than f . Theorem 5 describes ∆ ( f , f ′ ) for f being the solution to kernel ridge regression and f ′ its invariant ( feature averaged ) version and shows that it is positive when the target is invariant . More specifically , let X ∼ µ where µ is G-invariant and Y = f∗ ( X ) + ξ with f∗ G-invariant and E [ ξ ] = 0 , E [ ξ2 ] = σ2 < ∞ . Let f be the solution to kernel ridge regression with kernel k and regularisation parameter ρ > 0 on n i.i.d . training examples { ( Xi , Yi ) ∼ ( X , Y ) : i = 1 , . . . , n } and let f ′ be its feature averaged version . Our main result , Theorem 5 , says that E [ ∆ ( f , f ′ ) ] ≥ σ 2 dimeff ( H⊥ ) + E ( √ nMk + ρ/ √ n ) 2 where Mk = supx k ( x , x ) < ∞ , E ≥ 0 describes the approximation errors and dimeff ( H⊥ ) is the effective dimension of the RKHS H⊥ . For an RKHS H with kernel k the effective dimension is defined by dimeff ( H ) = ∫ X k ( x , y ) 2 dµ ( x ) dµ ( y ) . where X = suppµ . We return to this quantity at various points in the paper . It is important to note that the use of the feature averaged predictor f ′ as a comparator is without loss of generality . Any other predictor f ′′ that has test risk not larger than f ′ would satisfy the above bound , simply because this means ∆ ( f ′ , f ′′ ) ≥ 0 so ∆ ( f , f ′′ ) = ∆ ( f , f ′ ) +∆ ( f ′ , f ′′ ) ≥ ∆ ( f , f ′ ) .1 Finally , for intuition , in Theorem 7 we specialise Theorem 5 to the linear setting and compute the bound exactly . Assumptions and technical conditions are given in Section 2 along with an outline of the ideas of Elesedy and Zaidi [ 8 ] on which we build . Related works are discussed in Section 5 . 2 Background and Preliminaries . In this section we provide a brief introduction to reproducing kernel Hilbert spaces ( RKHS ) and the ideas we borrow from Elesedy and Zaidi [ 8 ] . Throughout this paper , H with be an RKHS with kernel k. In Section 2.2 we state some topological and measurability assumptions that are needed for our proofs . These conditions are benign and the reader not interested in technicalities need take from Section 2.2 only that µ is G-invariant and that the kernel k is bounded and satisfies Eq . ( 1 ) . We defer proofs to the Supplementary Material . 1To be completely clear : if , for instance , it so happens that projecting the RKHS onto a space of invariant predictors before doing KRR gives lower test risk than projecting afterwards ( what we are calling feature averaging ) , then our result applies in that case too . 2.1 RKHS Basics . A Hilbert space is an inner product space that is complete with respect to the norm topology induced by the inner product . A reproducing kernel Hilbert space ( RKHS ) H is Hilbert space of real functions f : X → R on which the evaluation functional δx : H → R with δx [ f ] = f ( x ) is continuous ∀x ∈ X , or , equivalently is a bounded operator . The Riesz Representation Theorem tells us that there is a unique function kx ∈ H such that δx [ f ] = ⟨kx , f⟩H for any f ∈ H , where ⟨· , ·⟩H : H×H → R is the inner product on H. We identify the function k : X ×X → R with k ( x , y ) = ⟨kx , ky⟩H as the reproducing kernel of H. Using the inner product representation , one can see that k is positive-definite and symmetric . Conversely , the Moore-Aronszajn Theorem shows that for any positive-definite and symmetric function k , there is a unique RKHS with reproducing kernel k. In addition , any Hilbert space admitting a reproducing kernel is an RKHS . Finally , another characterisation of H is as the completion of the set of linear combinations of the form fc ( x ) = ∑n i=1 cik ( x , xi ) for c1 , . . . , cn ∈ R and x1 , . . . , xn ∈ X . For ( many ) more details , see [ 30 , Chapter 4 ] . 2.2 Technical Setup and Assumptions . Input Space , Group and Measure Let G be a compact2 , second countable , Hausdorff topological group with Haar measure λ ( see [ 12 , Theorem 2.27 ] ) . Let X be a non-empty Polish space admitting a finite , G-invariant Borel measure µ , with suppµ = X . We normalise µ ( X ) = λ ( G ) = 1 , the latter is possible because λ is a Radon measure . We assume that G has a measurable action on X that we will write as gx for g ∈ G , x ∈ X . A measurable action is one such that the map g : G × X → X is ( λ ⊗ µ ) -measurable . A function f : X → R is G-invariant if f ( gx ) = f ( x ) ∀x ∈ X ∀g ∈ G. Similarly , a measure µ on X is G-invariant if ∀g ∈ G and any µ-measurable B ⊂ X the pushforward of µ by the action of G equals µ , i.e . ( g∗µ ) ( B ) = µ ( B ) . This means that if X ∼ µ then gX ∼ µ ∀g ∈ G. We will make use of the fact that the Haar measure is G-invariant when G acts on itself by either left or right multiplication , the latter holding because G is compact . Up to normalisation , λ is the unique measure on G with this property . The Kernel and the RKHS Let k : X × X → R be a measurable kernel with RKHS H such that k ( · , x ) : X → R is continuous for any x ∈ X . Assume that supx∈X k ( x , x ) = Mk < ∞ and note that this implies that k is bounded since k ( x , x′ ) = ⟨kx , kx′⟩H ≤ ∥kx∥H∥kx′∥H = √ k ( x , x ) √ k ( x′ , x′ ) ≤ Mk . Every f ∈ H is µ-measurable , bounded and continuous by [ 30 , Lemmas 4.24 and 4.28 ] and in addition H is separable using [ 30 , Lemma 4.33 ] . These conditions allow the application of [ 30 , Theorem 4.26 ] to relate H to L2 ( X , µ ) in the proofs building towards Lemma 3 , given in the Supplementary Material . We assume that the kernel satisfies , for all x , y ∈ X , ∫ G k ( gx , y ) dλ ( g ) = ∫ G k ( x , gy ) dλ ( g ) . ( 1 ) Equation ( 1 ) is our main assumption and we will make frequent use of it . For Eq . ( 1 ) to hold , it is sufficient to have k ( gx , y ) equal to k ( x , gy ) or k ( x , g−1y ) , where the latter uses compactness ( hence unimodularity ) of G to change variables g ↔ g−1 . Highlighting two special cases : any inner product kernel k ( x , x′ ) = κ ( ⟨x , x′⟩ ) such that the action of G is unitary with respect to ⟨· , ·⟩ satisfies Eq . ( 1 ) , as does any stationary kernel k ( x , x′ ) = κ ( ∥x− x′∥ ) with norm that is preserved by G in the sense that ∥gx− gx′∥ = ∥x− x′∥ for any g ∈ G , x , x′ ∈ X . If the norm/inner product is Euclidean , then any orthogonal representation of G will have this property.3

Summary: The authors present a theoretical generalization gap based on the invariance incorporated by the kernel. An orbit-averaged functional is defined with a set of invariant transformations G on data to produce an operator which maps a function to another function that averages the functions of the transformed data. The functional is used as the projection operator to produce the set of orthogonal functions to the invariant functions, and the generalization gap has been derived for kernel ridge regression using the effective dimensionality of those orthogonal functions.