Kernel Functional Optimisation

1 Introduction . Kernel machines ( Hofmann et al. , 2008 ) generally work well with low-dimensional and small to medium-scaled data . In most kernel machines , the kernel function is chosen from the standard bag of popular kernels ( Genton , 2001 , Stein , 2015 ) such as Squared Exponential kernel ( SE ) , Matérn kernel and Periodic kernel , or a weighted combination thereof ( Aiolli and Donini , 2015 , Gönen and Alpaydın , 2011 , Rakotomamonjy et al. , 2007 ) . Recent developments ( Jang et al. , 2017 , Wilson and Adams , 2013 ) in kernel learning parameterise the kernel function to boost the expressiveness of the kernel . However , the expressiveness of such kernels remains limited by the chosen parametric form and thus they often fall short in providing the best kernel function for complex data distributions . There have been some early attempts to design an optimal non-parametric kernel to remove the limitations associated with the parametric forms . Ong et al . ( 2003 , 2005 ) proposed a hyperkernel framework by defining a Reproducing Kernel Hilbert Space ( RKHS ) on the space of kernels i.e. , a kernel on kernels to support kernel learning . They formulate a semidefinite programming ( Vandenberghe and Boyd , 1996 ) based optimisation problem using the representer theorem ( Steinwart and Christmann , 2008 , Vapnik , 1999 ) to find the best kernel . However , their method suffers from two key limitations : ( i ) their way of enforcing the positive definiteness property produces a restrictive search space , resulting in a sub-optimal solution , and ( ii ) the computational complexity of their method scales with the dataset size , making it infeasible for larger datasets . Benton et al . ( 2019 ) proposed Functional Kernel Learning ( FKL ) , which extends the function space view of the Gaussian Process ( GP ) for kernel learning . FKL uses a transformed GP over a spectral density to define a distribution over kernels . However , the formulation of kernel functionals using the spectral densities induces strong assumptions on the properties such as periodicity , stationarity , etc . and thus are not generally applicable . Malkomes et al . ( 2016 ) proposed an automated kernel selection ( BOMS ) using Bayesian optimisation . The kernel space in BOMS is defined by the base kernels and the associated grammar to combine them . Although the search space is constructed by summing or multiplying the base kernels , the resultant kernel space is restricted in the compositional space of parametric forms . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this paper , we propose a generic framework called Kernel Functional Optimisation ( KFO ) to address the aforesaid shortcomings . First , it provides a flexible form of kernel learning whose computational complexity is decoupled from dataset size . Next , it allows us to use a computationally efficient Bayesian optimisation method to find the best kernel . We incorporate hyperkernels into our Bayesian framework that allows us to search for the optimal kernel in a Hilbert space of kernels spanned by the hyperkernel ( Ong et al. , 2005 ) . We draw kernel functionals from a ( hyper ) GP distribution fitted using a hyperkernel . As the kernel drawn from the hyper-GP may be indefinite , we provide ways to ensure positive definiteness by transforming indefinite , or Kreı̆n ( Oglic and Gärtner , 2019 , Ong et al. , 2004 ) kernel space into a positive definite kernel space . The optimisation of kernel functionals necessitates solving larger covariance matrices and thus adds to the computational burden of the overall process . To speed up the computations , we perform a low-rank decomposition of the covariance matrix . Further , we provide a theoretical analysis of our method showing that it converges efficiently as in its cumulative regret grows only sub-linearly and eventually vanishes . We evaluate the performance of our method on both synthetic and real-world datasets using SVM classification ( Diehl and Cauwenberghs , 2003 , Scholkopf and Smola , 2001 , Burges , 1998 ) and GP regression tasks . Comparison of predictive performance against the state-of-the-art baselines demonstrates the superiority of our method . Further , we compare with the state-of-the-art performance reported in the latest survey paper on classifier comparison ( Zhang et al. , 2017 ) and find that our method provides the best performance on most of the datasets . Our main contributions in this paper are as follows : ( i ) we propose a novel approach for finding the best non-parametric kernel using hyperkernels and Bayesian functional optimisation ( Section 3 ) , ( ii ) we provide methods to ensure positive definiteness of the kernels optimised ( Section 3 ) , ( iii ) we derive the convergence guarantees to demonstrate that the regret grows sub-linearly for our proposed method ( Section 4 ) , ( iv ) we provide empirical results on both synthetic and real-world datasets to prove the usefulness ( Section 5 ) . 2 Background . Notations We use lower case bold fonts v for vectors and vi for each element in v. vᵀ is the transpose . We use upper case bold fonts M ( and bold greek symbols ) for matrices and Mij for each element in M. | · | for the absolute value . Nn = { 1 , 2 , · · · , n } . R for Reals . X is a non-empty ( index ) set and x ∈ X . X̃ is a non-empty ( compounded index ) set and x̃ ∈ X̃ , X̃ = X 2 . ( · ) + clips a negative value to zero . J·K is the Iverson bracket ( Iverson , 1962 ) defined for any boolean value I as JIK = 1 , if I is True , 0 otherwise . Matrix M = [ Mij ] i , j∈N and ‖M‖F is the Frobenius Norm of M . 2.1 Bayesian Optimisation . Bayesian Optimisation ( BO ) ( Brochu et al. , 2010 , Shahriari et al. , 2015 , Frazier , 2018 ) offers an elegant framework for finding the global extrema of an unknown , expensive and noisy function f ( x ) , represented as x∗ = argmaxx∈X f ( x ) , where X is a compact search space . Bayesian optimisation is comprised of two main components : ( i ) a Gaussian Process ( GP ) ( Williams and Rasmussen , 2006 ) model of f , and ( ii ) an acquisition function ( u ) ( Kushner , 1964 , Močkus , 1975 , Wilson et al. , 2018 ) to guide optimisation . Let D = { x1 : t , y1 : t } denote a set of observations of f , where y = f ( x ) + ′ is the noisy observation corrupted with white Gaussian noise ′ ∈ N ( 0 , σ2noise ) . Then the predictive distribution at any point x∗ is given as f ( x∗ ) |D ∼ N ( µ ( x∗ ) , σ2 ( x∗ ) ) , where µ ( x∗ ) = kᵀ [ K + σ2noiseI ] −1y1 : t , σ2 ( x∗ ) = k ( x∗ , x∗ ) − kᵀ [ K + σ2noiseI ] −1k , k = [ k ( x∗ , x1 ) · · · k ( x∗ , xt ) ] , k : X × X → R and K = [ k ( xi , xj ) ] i , j∈Nt . The negative log-likelihood for a GP distribution is − logP ( y∗|D , x∗ ) = 12 log ( 2πσ2 ( x∗ ) ) + ( y∗−µ ( x∗ ) ) 2 2σ2 ( x∗ ) ( 1 ) The acquisition function ( u ) guides the search by balancing between exploitation ( searching known high-value regions ) and exploration ( searching high-variance regions ) . Gaussian Process - Upper Confidence Bound ( GP-UCB ) acquisition function ( Srinivas et al. , 2012 , Brochu et al. , 2010 ) is the commonly used acquisition function to find the next best candidate for the evaluation , given as ut ( x ) = µ ( x ) + √ βt σ ( x ) ( 2 ) where βt grows as O ( log t ) with iteration t. Further , it can be shown that the average regret ( R , 1t ∑t t′=1 |f ( x∗ ) − f ( xt′ ) | ) grows as O ( √ log t/t ) , and hence the average regret vanishes as t→∞ . An algorithm for standard Bayesian optimisation is provided in the supplementary material . The aforementioned standard Bayesian optimisation procedure often suffers from scaling issues originating from the curse of dimensionality . Wang et al . ( 2016 ) proposed REMBO - Random EMbedding Bayesian Optimisation - to address these scaling issues . REMBO works by projecting the objective function onto a lower-dimensional subspace prior to optimisation . LINEBO ( Kirschner et al. , 2019 ) builds on the same idea but instead of a fixed subspace , it decomposes the given black-box optimisation problem into a sequence of one-dimensional subproblems . Further , our method builds upon the principles of Bayesian functional optimisation methodologies ( Vien et al. , 2018 , Vellanki et al. , 2019 , Shilton et al. , 2020 ) in the literature to find a function to optimise the given process . 2.2 RKHS and Hyper-RKHS . The kernel functions used in the Gaussian process uniquely define an associated Reproducing Kernel Hilbert Space ( RKHS ) ( Aronszajn , 1950 ) . Formally : Definition 1 : LetHk be a Hilbert space of functions f : X → R on a non-empty set X . A function k : X × X → R is a reproducing kernel of Hk , and Hk a Reproducing Kernel Hilbert Space ( RKHS ) , if the following properties are satisfied . • k spansHk i.e. , Hk = span { k ( · , x ) |x ∈ X } • ∀x ∈ X , ∀f ∈ Hk , 〈f ( · ) , k ( · , x ) 〉Hk = f ( x ) ( the reproducing property ) • ∀x , x′ ∈ X , k ( x , x′ ) = 〈k ( · , x ) , k ( · , x′ ) 〉Hk Next , we consider the Reproducing Kernel Hilbert Space ( RKHS ) of kernels by introducing a compounded index set X̃ : X × X and a hyperkernel κ ( Ong and Smola , 2003 , Ong et al. , 2003 ) . Analogous to the RKHS ( Aronszajn , 1950 ) associated with the kernel function , a hyperkernel defines an associated Hyper-Reproducing Kernel Hilbert Space ( Hyper-RKHS ) ( Ong et al. , 2003 ) . Definition 2 : Let X be a non-empty set and X̃ denote X × X . The Hilbert space Hκ of functions k : X̃ → R is called a Hyper-Reproducing Kernel Hilbert Space ( Hyper-RKHS ) , if there exists a hyperkernel κ : X̃ × X̃ → R that satisfies the following properties : • κ spansHκ i.e. , Hκ = span { κ ( · , x̃ ) | x̃ ∈ X̃ } • ∀x̃ ∈ X̃ , ∀k ∈ Hκ , 〈k ( · ) , κ ( · , x̃ ) 〉Hκ = k ( x̃ ) ( the reproducing property ) • ∀x̃ , x̃′ ∈ X̃ , κ ( x̃ , x̃′ ) = 〈κ ( · , x̃ ) , κ ( · , x̃′ ) 〉Hκ • κ ( x′ , x′′ , x′′′ , x′′′′ ) = κ ( x′′ , x′ , x′′′ , x′′′′ ) ∀x′ , x′′ , x′′′ , x′′′′∈X The GP distribution defined by a hyperkernel κ is a distribution on the space of kernels . This Hyper-RKHS is a Hilbert space comprised of positive definite , negative definite and indefinite kernels . A Kreı̆n kernel k ( Oglic and Gärtner , 2018 , Ong et al. , 2004 ) is an indefinite kernel with a positive decomposition i.e. , there exist positive kernels k+ ∈ H+ and k− ∈ H− , such that k = k+ − k− . From Definition 2 , we see that κ ( x̃ , x̃′ ) = κ ( x′ , x′′ , x′′′ , x′′′′ ) is a kernel , where x̃ = ( x′ , x′′ ) . Generally , the samples drawn from GP ( 0 , k ) do not lie in the corresponding RKHS Hk , but in a larger RKHSHk′ 6=k ( see discussion in Kanagawa et al . ( 2018 ) , Remark 3.8 and Section 4 ) . We also note that the posterior mean of GP ( 0 , k ) lies in the RKHS Hk . Similarly , with hyperGP , the samples drawn from GPκ ( 0 , κ ) lie in RKHS Hκ′ 6=κ , whereas its posterior mean ( µ ) lies in Hκ . Further , µ can be decomposed with positive and negative weights as µ = µ+ − µ− =∑ i αi+κ ( · , x̃i+ ) − ∑ i αi−κ ( · , x̃i− ) , where αi+ , αi− > 0 ; and µ± = ∑ i αi±κ ( · , x̃i± ) is a kernel ( Definition 2 and Ong et al . ( 2004 ) ) . Thus , µ = µ+−µ− is a Kreı̆n kernel ( Oglic and Gärtner , 2019 ) .

Summary: This article proposes to use the hyperkernel formalism coupled with Gaussian Processes to derive the ‘best’ kernel for regression. The latter kernel depends mostly on the observations rather than on a pre-selected kernel family. The proposed procedure relies on several stages of Bayesian optimization as well as a clipping of the coefficients to avoid dealing with (indefinite) Krein kernels. The performance is presented on both synthetic and benchmark datasets, to assess visually the fit of the confidence intervals as well as the numerical performance.