Kernel Functional Optimisation

1 Introduction . Kernel machines ( Hofmann et al. , 2008 ) generally work well with low-dimensional and small to medium-scaled data . In most kernel machines , the kernel function is chosen from the standard bag of popular kernels ( Genton , 2001 , Stein , 2015 ) such as Squared Exponential kernel ( SE ) , Matérn kernel and Periodic kernel , or a weighted combination thereof ( Aiolli and Donini , 2015 , Gönen and Alpaydın , 2011 , Rakotomamonjy et al. , 2007 ) . Recent developments ( Jang et al. , 2017 , Wilson and Adams , 2013 ) in kernel learning parameterise the kernel function to boost the expressiveness of the kernel . However , the expressiveness of such kernels remains limited by the chosen parametric form and thus they often fall short in providing the best kernel function for complex data distributions . There have been some early attempts to design an optimal non-parametric kernel to remove the limitations associated with the parametric forms . Ong et al . ( 2003 , 2005 ) proposed a hyperkernel framework by defining a Reproducing Kernel Hilbert Space ( RKHS ) on the space of kernels i.e. , a kernel on kernels to support kernel learning . They formulate a semidefinite programming ( Vandenberghe and Boyd , 1996 ) based optimisation problem using the representer theorem ( Steinwart and Christmann , 2008 , Vapnik , 1999 ) to find the best kernel . However , their method suffers from two key limitations : ( i ) their way of enforcing the positive definiteness property produces a restrictive search space , resulting in a sub-optimal solution , and ( ii ) the computational complexity of their method scales with the dataset size , making it infeasible for larger datasets . Benton et al . ( 2019 ) proposed Functional Kernel Learning ( FKL ) , which extends the function space view of the Gaussian Process ( GP ) for kernel learning . FKL uses a transformed GP over a spectral density to define a distribution over kernels . However , the formulation of kernel functionals using the spectral densities induces strong assumptions on the properties such as periodicity , stationarity , etc . and thus are not generally applicable . Malkomes et al . ( 2016 ) proposed an automated kernel selection ( BOMS ) using Bayesian optimisation . The kernel space in BOMS is defined by the base kernels and the associated grammar to combine them . Although the search space is constructed by summing or multiplying the base kernels , the resultant kernel space is restricted in the compositional space of parametric forms . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this paper , we propose a generic framework called Kernel Functional Optimisation ( KFO ) to address the aforesaid shortcomings . First , it provides a flexible form of kernel learning whose computational complexity is decoupled from dataset size . Next , it allows us to use a computationally efficient Bayesian optimisation method to find the best kernel . We incorporate hyperkernels into our Bayesian framework that allows us to search for the optimal kernel in a Hilbert space of kernels spanned by the hyperkernel ( Ong et al. , 2005 ) . We draw kernel functionals from a ( hyper ) GP distribution fitted using a hyperkernel . As the kernel drawn from the hyper-GP may be indefinite , we provide ways to ensure positive definiteness by transforming indefinite , or Kreı̆n ( Oglic and Gärtner , 2019 , Ong et al. , 2004 ) kernel space into a positive definite kernel space . The optimisation of kernel functionals necessitates solving larger covariance matrices and thus adds to the computational burden of the overall process . To speed up the computations , we perform a low-rank decomposition of the covariance matrix . Further , we provide a theoretical analysis of our method showing that it converges efficiently as in its cumulative regret grows only sub-linearly and eventually vanishes . We evaluate the performance of our method on both synthetic and real-world datasets using SVM classification ( Diehl and Cauwenberghs , 2003 , Scholkopf and Smola , 2001 , Burges , 1998 ) and GP regression tasks . Comparison of predictive performance against the state-of-the-art baselines demonstrates the superiority of our method . Further , we compare with the state-of-the-art performance reported in the latest survey paper on classifier comparison ( Zhang et al. , 2017 ) and find that our method provides the best performance on most of the datasets . Our main contributions in this paper are as follows : ( i ) we propose a novel approach for finding the best non-parametric kernel using hyperkernels and Bayesian functional optimisation ( Section 3 ) , ( ii ) we provide methods to ensure positive definiteness of the kernels optimised ( Section 3 ) , ( iii ) we derive the convergence guarantees to demonstrate that the regret grows sub-linearly for our proposed method ( Section 4 ) , ( iv ) we provide empirical results on both synthetic and real-world datasets to prove the usefulness ( Section 5 ) . 2 Background . Notations We use lower case bold fonts v for vectors and vi for each element in v. vᵀ is the transpose . We use upper case bold fonts M ( and bold greek symbols ) for matrices and Mij for each element in M. | · | for the absolute value . Nn = { 1 , 2 , · · · , n } . R for Reals . X is a non-empty ( index ) set and x ∈ X . X̃ is a non-empty ( compounded index ) set and x̃ ∈ X̃ , X̃ = X 2 . ( · ) + clips a negative value to zero . J·K is the Iverson bracket ( Iverson , 1962 ) defined for any boolean value I as JIK = 1 , if I is True , 0 otherwise . Matrix M = [ Mij ] i , j∈N and ‖M‖F is the Frobenius Norm of M . 2.1 Bayesian Optimisation . Bayesian Optimisation ( BO ) ( Brochu et al. , 2010 , Shahriari et al. , 2015 , Frazier , 2018 ) offers an elegant framework for finding the global extrema of an unknown , expensive and noisy function f ( x ) , represented as x∗ = argmaxx∈X f ( x ) , where X is a compact search space . Bayesian optimisation is comprised of two main components : ( i ) a Gaussian Process ( GP ) ( Williams and Rasmussen , 2006 ) model of f , and ( ii ) an acquisition function ( u ) ( Kushner , 1964 , Močkus , 1975 , Wilson et al. , 2018 ) to guide optimisation . Let D = { x1 : t , y1 : t } denote a set of observations of f , where y = f ( x ) + ′ is the noisy observation corrupted with white Gaussian noise ′ ∈ N ( 0 , σ2noise ) . Then the predictive distribution at any point x∗ is given as f ( x∗ ) |D ∼ N ( µ ( x∗ ) , σ2 ( x∗ ) ) , where µ ( x∗ ) = kᵀ [ K + σ2noiseI ] −1y1 : t , σ2 ( x∗ ) = k ( x∗ , x∗ ) − kᵀ [ K + σ2noiseI ] −1k , k = [ k ( x∗ , x1 ) · · · k ( x∗ , xt ) ] , k : X × X → R and K = [ k ( xi , xj ) ] i , j∈Nt . The negative log-likelihood for a GP distribution is − logP ( y∗|D , x∗ ) = 12 log ( 2πσ2 ( x∗ ) ) + ( y∗−µ ( x∗ ) ) 2 2σ2 ( x∗ ) ( 1 ) The acquisition function ( u ) guides the search by balancing between exploitation ( searching known high-value regions ) and exploration ( searching high-variance regions ) . Gaussian Process - Upper Confidence Bound ( GP-UCB ) acquisition function ( Srinivas et al. , 2012 , Brochu et al. , 2010 ) is the commonly used acquisition function to find the next best candidate for the evaluation , given as ut ( x ) = µ ( x ) + √ βt σ ( x ) ( 2 ) where βt grows as O ( log t ) with iteration t. Further , it can be shown that the average regret ( R , 1t ∑t t′=1 |f ( x∗ ) − f ( xt′ ) | ) grows as O ( √ log t/t ) , and hence the average regret vanishes as t→∞ . An algorithm for standard Bayesian optimisation is provided in the supplementary material . The aforementioned standard Bayesian optimisation procedure often suffers from scaling issues originating from the curse of dimensionality . Wang et al . ( 2016 ) proposed REMBO - Random EMbedding Bayesian Optimisation - to address these scaling issues . REMBO works by projecting the objective function onto a lower-dimensional subspace prior to optimisation . LINEBO ( Kirschner et al. , 2019 ) builds on the same idea but instead of a fixed subspace , it decomposes the given black-box optimisation problem into a sequence of one-dimensional subproblems . Further , our method builds upon the principles of Bayesian functional optimisation methodologies ( Vien et al. , 2018 , Vellanki et al. , 2019 , Shilton et al. , 2020 ) in the literature to find a function to optimise the given process . 2.2 RKHS and Hyper-RKHS . The kernel functions used in the Gaussian process uniquely define an associated Reproducing Kernel Hilbert Space ( RKHS ) ( Aronszajn , 1950 ) . Formally : Definition 1 : LetHk be a Hilbert space of functions f : X → R on a non-empty set X . A function k : X × X → R is a reproducing kernel of Hk , and Hk a Reproducing Kernel Hilbert Space ( RKHS ) , if the following properties are satisfied . • k spansHk i.e. , Hk = span { k ( · , x ) |x ∈ X } • ∀x ∈ X , ∀f ∈ Hk , 〈f ( · ) , k ( · , x ) 〉Hk = f ( x ) ( the reproducing property ) • ∀x , x′ ∈ X , k ( x , x′ ) = 〈k ( · , x ) , k ( · , x′ ) 〉Hk Next , we consider the Reproducing Kernel Hilbert Space ( RKHS ) of kernels by introducing a compounded index set X̃ : X × X and a hyperkernel κ ( Ong and Smola , 2003 , Ong et al. , 2003 ) . Analogous to the RKHS ( Aronszajn , 1950 ) associated with the kernel function , a hyperkernel defines an associated Hyper-Reproducing Kernel Hilbert Space ( Hyper-RKHS ) ( Ong et al. , 2003 ) . Definition 2 : Let X be a non-empty set and X̃ denote X × X . The Hilbert space Hκ of functions k : X̃ → R is called a Hyper-Reproducing Kernel Hilbert Space ( Hyper-RKHS ) , if there exists a hyperkernel κ : X̃ × X̃ → R that satisfies the following properties : • κ spansHκ i.e. , Hκ = span { κ ( · , x̃ ) | x̃ ∈ X̃ } • ∀x̃ ∈ X̃ , ∀k ∈ Hκ , 〈k ( · ) , κ ( · , x̃ ) 〉Hκ = k ( x̃ ) ( the reproducing property ) • ∀x̃ , x̃′ ∈ X̃ , κ ( x̃ , x̃′ ) = 〈κ ( · , x̃ ) , κ ( · , x̃′ ) 〉Hκ • κ ( x′ , x′′ , x′′′ , x′′′′ ) = κ ( x′′ , x′ , x′′′ , x′′′′ ) ∀x′ , x′′ , x′′′ , x′′′′∈X The GP distribution defined by a hyperkernel κ is a distribution on the space of kernels . This Hyper-RKHS is a Hilbert space comprised of positive definite , negative definite and indefinite kernels . A Kreı̆n kernel k ( Oglic and Gärtner , 2018 , Ong et al. , 2004 ) is an indefinite kernel with a positive decomposition i.e. , there exist positive kernels k+ ∈ H+ and k− ∈ H− , such that k = k+ − k− . From Definition 2 , we see that κ ( x̃ , x̃′ ) = κ ( x′ , x′′ , x′′′ , x′′′′ ) is a kernel , where x̃ = ( x′ , x′′ ) . Generally , the samples drawn from GP ( 0 , k ) do not lie in the corresponding RKHS Hk , but in a larger RKHSHk′ 6=k ( see discussion in Kanagawa et al . ( 2018 ) , Remark 3.8 and Section 4 ) . We also note that the posterior mean of GP ( 0 , k ) lies in the RKHS Hk . Similarly , with hyperGP , the samples drawn from GPκ ( 0 , κ ) lie in RKHS Hκ′ 6=κ , whereas its posterior mean ( µ ) lies in Hκ . Further , µ can be decomposed with positive and negative weights as µ = µ+ − µ− =∑ i αi+κ ( · , x̃i+ ) − ∑ i αi−κ ( · , x̃i− ) , where αi+ , αi− > 0 ; and µ± = ∑ i αi±κ ( · , x̃i± ) is a kernel ( Definition 2 and Ong et al . ( 2004 ) ) . Thus , µ = µ+−µ− is a Kreı̆n kernel ( Oglic and Gärtner , 2019 ) .

Summary: The paper proposes a zero-order optimization method where the optimized variable is a kernel function in Hyper-RKHS induced by a selected hyper-kernel. The method is an instance of the Bayesian Optimization method LINEBO [Kirschner 2019]. The contribution is in the adaptation of the LINEBO algorithm for efficient optimization w.r.t. positive definite kernel functions. The algorithm is applied to the optimization of kernel functions for C-SVM and GP regression. Experiments show significant improvement over existing methods.