Dual Parameterization of Sparse Variational Gaussian Processes

1 Introduction . Gaussian processes ( GPs , [ 31 ] ) have become ubiquitous models in the probabilistic machine learning toolbox , but their application is challenging due to two issues : poor O ( n3 ) scaling in the number of data points , n , and challenging approximate inference in non-conjugate ( non-Gaussian ) models . In recent years , variational inference has become the go-to solution to overcome these problems , where sparse variational GP methods [ 35 ] tackle both the non-conjugacy and high computation cost . The computation is reduced to O ( nm2 ) by using a small number of m≪ n inducing-input locations . For large problems , the SVGP framework is a popular choice as it enables fast stochastic training , and reduces the cost to O ( m3 + nbm 2 ) per step , where nb is the batch size [ 10 , 11 ] . It is a common practice in SVGP to utilize the standard mean-covariance parameterization which requiresO ( m2 ) memory . Inference is carried out by optimizing an objectiveL ( q ) that uses a Gaussian distribution q parameterized by parameters ξ = ( m , L ) where m is the mean and L is the Cholesky factor of the covariance matrix . We will refer to the SVGP methods using such parameterization as the q-SVGP methods . The advantage of this formulation is that , for log-concave likelihoods , the objective is convex and gradient-based optimization works well [ 2 ] . The optimization can further be improved by using natural gradient descent ( NGD ) which is shown to be less sensitive to learning rates [ 33 ] . The NGD algorithm with q-SVGP parameterization is currently the state-of-the-art and available in the existing software implementations such as GPflow [ 26 ] and GPyTorch [ 9 ] . An alternate parameterization to the q-SVGP parameterization is the one where every likelihood is assigned two sets of parameters which require O ( n ) memory . Existence of such parameterizations was initially shown by Csató and Opper [ 8 ] for general GP models , and later on extended to variational ∗Both authors contributed equally . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . objectives for GPs [ 28 , 29 ] , and also to latent Gaussian models by using Lagrangian duality [ 20 , 16 ] . Due to this later connection , we refer to this parameterization as the dual parameterization where the parameters are the Lagrange multipliers , ensuring that the marginal mean and variance of each latent function is consistent to the marginals obtained by the full GP [ 16 ] . Expectation propagation ( EP , [ 27 ] ) too naturally employ such parameterizations , but by using site parameters . Although unrelated to duality , such methods are popular for GP inference , and due to this connection , we will refer to the methods using dual parameterization as t-SVGP ( the letter ‘ t ’ refers to the sites ) . To the best of our knowledge , the dual parameterization for SVGP has only been used to speed up computation and inference for the specific case of Markovian GPs [ 4 , 38 ] . The main contribution of this work is to introduce the dual parameterization for SVGP and show that it speeds up both the learning and inference . For inference , we show that the dual parameters are automatically obtained through a different formulation of NGD , written in terms of the expectation parameters [ 17 , 15 , 18 ] . The formulation is fast since it avoids the use of sluggish automatic differentiation to compute the natural gradients . We also match the typicalO ( m2 ) memory complexity in other SVGP methods by introducing a tied parametrization . For learning , we show that the dual parameterization results in a tighter lower bound to the marginal likelihood , which speed-up the hyperparameter optimization ( see Fig . 1 ) . We provide extensive evaluation on benchmark data sets , which confirms our findings . Our work attempts to revive the dual parameterization , which was popular in the early 2000s , but was somehow forgotten and not used in the recent SVGP algorithms . 2 Background : Variational Inference for Gaussian Processes Models . Gaussian processes ( GPs , [ 31 ] ) are distributions over functions , commonly used in machine learning to endow latent functions in generative models with rich and interpretable priors . These priors can provide strong inductive biases for regression tasks in the small data regime . GP-based models are the ones that employ a GP prior over the ( latent ) functions f ( · ) ∼ GP ( µ ( · ) , κ ( · , ·′ ) ) , where the prior is completely characterized by the mean function µ ( · ) and covariance function κ ( · , ·′ ) . We denote f ( · ) as a function but occasionally simplify the notation to just f in the interest of reducing clutter . Given a data set D = ( X , y ) = { ( xi , yi ) } n i=1 of input–output pairs , we denote by f the vector of function evaluations at the inputs { f ( xi ) } n i=1 . The function evaluation at xi are passed through likelihood functions to model the outputs y ∈ R , e.g . by specifying p ( y | f ) : = ∏n i=1 p ( yi | fi ) . Prediction at a new test input x∗ is obtained by computing the distribution p ( f ( x∗ ) | D , x∗ ) . For Gaussian likelihoods N ( yi | fi , σ 2 ) , the predictive distribution is available in closed form as a Gaussian distribution N ( f ( x∗ ) |mGPR ( x∗ ) , vGPR ( x∗ ) ) with mean and variance defined as follows : mGPR ( x∗ ) : = k ⊤ f∗ ( Kff + σ 2In ) −1y , and vGPR ( x∗ ) : = κ∗∗ − k ⊤ f∗ ( Kff + σ 2In ) −1kf∗ , ( 1 ) where kf∗ is a vector of κ ( x∗ , xi ) as the i th element for all xi ∈ X , Kff is an n × n matrix with κ ( xi , xj ) as the ij th entry , and κ∗∗ = κ ( x∗ , x∗ ) . 2.1 Variational Expectation–Maximization for GPs with Non-Conjugate Likelihoods . For non-Gaussian likelihoods , the posterior and predictive distributions are no longer Gaussian , and we need to resort to approximate inference methods . Variational inference is a popular choice because it allows for fast posterior approximation and hyperparameter learning via stochastic training [ 35 , 10 ] . Denoting kernel hyperparameters by θ and the corresponding GP prior by pθ ( f ) , the posterior distribution can be written as pθ ( f |y ) = pθ ( f ) p ( y | f ) /pθ ( y ) , where pθ ( y ) = ∫ pθ ( f , y ) df is the marginal likelihood of the observations . We seek to approximate pθ ( f |y ) ≈ qf ( f ) by a Gaussian distribution whose parameters can be obtained by optimizing the following evidence lower bound ( ELBO ) to the log-marginal likelihood , log pθ ( y ) ≥ Lq ( qf , θ ) = ∑n i=1 Eqf ( fi ) [ log p ( yi | fi ) ] −DKL [ qf ( f ) ‖ pθ ( f ) ] . ( 2 ) For the variational approximation , it is a standard practice to choose the mean-covariance parameterization , denoted by ξ = ( m , S ) . It is also common to use the Cholesky factor L instead [ 3 ] since it is uniquely determined for a covariance matrix . The multivariate normal distribution is part of the exponential family [ 37 ] , i.e . it ’ s probability density function take the form p ( x ) = exp ( η⊤T ( x ) − a ( η ) ) , with natural parameters η = ( S−1m , −S−1/2 ) and sufficient statistics T ( x ) = [ x , xx⊤ ] . This natural parameterization is also a common choice , along with the associated expectation parameterization µ = Eq [ T ( x ) ] = ( m , S+mm ⊤ ) . A final choice of the parameterization , called the whitened parameterization [ 36 ] , uses a variable v ∼ N ( v ; mv , Sv ) along with the transformation f = Lv , to parameterize m = Lmv and S = LSvL ⊤ . In the following , we will consider the ELBO with several parameterizations and , to make the notation clearer , we will indicate the parameterization used with a subscript with L. For example , we may have Lξ ( ξ , θ ) = Lη ( η , θ ) = Lµ ( µ , θ ) , which are all clearly equal due to a unique mapping between the parameterizations ; see [ 25 ] for more details on the maps . The ELBO can be optimized by using a variational expectation–maximization ( VEM ) procedure , where we alternate between optimizing variational parameters , say ξ , and hyperparameters θ , E-step : ξ∗t ← argmaxξ Lξ ( ξ , θt ) , M-step : θt+1 ← argmaxθ Lξ ( ξ ∗ t , θ ) , ( 3 ) where t denotes the iterations , and we have explicitly written the dependence of optimal parameter ξ∗ ( θt ) as a function of the old parameter θt . Both and E and M-steps can be carried out with gradient descent , for example , using an iteration of the form ξ ( k+1 ) t ← ξ ( k ) t + ρk∇ξLξ ( ξ ( k ) t , θt ) for E-step , which would ultimately converge to ξ∗t . A similar iterative method can be used for the M-step . 2.2 Inference via Natural-Gradient Descent ( NGD ) . A popular strategy for the E-step is to use natural gradient descent where we replace the gradient by the one preconditioned using the Fisher information matrix F ( ξ ) of qf ( f ) . We denote natural gradients by ∇̃ξLξ ( ξ , θ ) = F ( ξ ) −1∇ξLξ ( ξ , θ ) , to get the following update , ξ ( k+1 ) t ← ξ ( k ) t + ρk∇̃ξLξ ( ξ ( k ) t , θt ) . ( 4 ) Such updates can converge faster than gradient descent [ 33 , 17 , 18 ] , and at times are less sensitive to the choice of the learning rate ρk due to the scaling with the Fisher information matrix . The implementation simplifies greatly when using natural parameterizations , η ( k+1 ) t ← η ( k ) t + ρk∇µLµ ( µ ( k ) t , θt ) , ( 5 ) because ∇̃ηLη ( η , θ ) = ∇µLµ ( µ , θ ) , that is , the natural gradients with respect to η are in fact the gradients with respect to the expectation parameter µ [ 18 ] . In the remainder of the paper we will frequently use this property , and refer to the natural gradient with respect to η by the gradients with respect to expectation parameterization µ . The VEM procedure with NGD has recently become a popular choice for sparse variants of GPs , which we explain next .

Summary: The authors argue for a different (existing) parametrization for training (non-conjugate) Gaussian Processes. They show that this parametrization can be interpreted as the optimal result of a dual problem, and that one can compute easily natural gradient updates. Additionally this parametrization "may" lead to a tighter ELBO because of a stronger dependency on the hyperparameters (kernel parameters). Experiments are run to show the improvement over the standard "q-SVGP".