Dual Parameterization of Sparse Variational Gaussian Processes

1 Introduction . Gaussian processes ( GPs , [ 31 ] ) have become ubiquitous models in the probabilistic machine learning toolbox , but their application is challenging due to two issues : poor O ( n3 ) scaling in the number of data points , n , and challenging approximate inference in non-conjugate ( non-Gaussian ) models . In recent years , variational inference has become the go-to solution to overcome these problems , where sparse variational GP methods [ 35 ] tackle both the non-conjugacy and high computation cost . The computation is reduced to O ( nm2 ) by using a small number of m≪ n inducing-input locations . For large problems , the SVGP framework is a popular choice as it enables fast stochastic training , and reduces the cost to O ( m3 + nbm 2 ) per step , where nb is the batch size [ 10 , 11 ] . It is a common practice in SVGP to utilize the standard mean-covariance parameterization which requiresO ( m2 ) memory . Inference is carried out by optimizing an objectiveL ( q ) that uses a Gaussian distribution q parameterized by parameters ξ = ( m , L ) where m is the mean and L is the Cholesky factor of the covariance matrix . We will refer to the SVGP methods using such parameterization as the q-SVGP methods . The advantage of this formulation is that , for log-concave likelihoods , the objective is convex and gradient-based optimization works well [ 2 ] . The optimization can further be improved by using natural gradient descent ( NGD ) which is shown to be less sensitive to learning rates [ 33 ] . The NGD algorithm with q-SVGP parameterization is currently the state-of-the-art and available in the existing software implementations such as GPflow [ 26 ] and GPyTorch [ 9 ] . An alternate parameterization to the q-SVGP parameterization is the one where every likelihood is assigned two sets of parameters which require O ( n ) memory . Existence of such parameterizations was initially shown by Csató and Opper [ 8 ] for general GP models , and later on extended to variational ∗Both authors contributed equally . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . objectives for GPs [ 28 , 29 ] , and also to latent Gaussian models by using Lagrangian duality [ 20 , 16 ] . Due to this later connection , we refer to this parameterization as the dual parameterization where the parameters are the Lagrange multipliers , ensuring that the marginal mean and variance of each latent function is consistent to the marginals obtained by the full GP [ 16 ] . Expectation propagation ( EP , [ 27 ] ) too naturally employ such parameterizations , but by using site parameters . Although unrelated to duality , such methods are popular for GP inference , and due to this connection , we will refer to the methods using dual parameterization as t-SVGP ( the letter ‘ t ’ refers to the sites ) . To the best of our knowledge , the dual parameterization for SVGP has only been used to speed up computation and inference for the specific case of Markovian GPs [ 4 , 38 ] . The main contribution of this work is to introduce the dual parameterization for SVGP and show that it speeds up both the learning and inference . For inference , we show that the dual parameters are automatically obtained through a different formulation of NGD , written in terms of the expectation parameters [ 17 , 15 , 18 ] . The formulation is fast since it avoids the use of sluggish automatic differentiation to compute the natural gradients . We also match the typicalO ( m2 ) memory complexity in other SVGP methods by introducing a tied parametrization . For learning , we show that the dual parameterization results in a tighter lower bound to the marginal likelihood , which speed-up the hyperparameter optimization ( see Fig . 1 ) . We provide extensive evaluation on benchmark data sets , which confirms our findings . Our work attempts to revive the dual parameterization , which was popular in the early 2000s , but was somehow forgotten and not used in the recent SVGP algorithms . 2 Background : Variational Inference for Gaussian Processes Models . Gaussian processes ( GPs , [ 31 ] ) are distributions over functions , commonly used in machine learning to endow latent functions in generative models with rich and interpretable priors . These priors can provide strong inductive biases for regression tasks in the small data regime . GP-based models are the ones that employ a GP prior over the ( latent ) functions f ( · ) ∼ GP ( µ ( · ) , κ ( · , ·′ ) ) , where the prior is completely characterized by the mean function µ ( · ) and covariance function κ ( · , ·′ ) . We denote f ( · ) as a function but occasionally simplify the notation to just f in the interest of reducing clutter . Given a data set D = ( X , y ) = { ( xi , yi ) } n i=1 of input–output pairs , we denote by f the vector of function evaluations at the inputs { f ( xi ) } n i=1 . The function evaluation at xi are passed through likelihood functions to model the outputs y ∈ R , e.g . by specifying p ( y | f ) : = ∏n i=1 p ( yi | fi ) . Prediction at a new test input x∗ is obtained by computing the distribution p ( f ( x∗ ) | D , x∗ ) . For Gaussian likelihoods N ( yi | fi , σ 2 ) , the predictive distribution is available in closed form as a Gaussian distribution N ( f ( x∗ ) |mGPR ( x∗ ) , vGPR ( x∗ ) ) with mean and variance defined as follows : mGPR ( x∗ ) : = k ⊤ f∗ ( Kff + σ 2In ) −1y , and vGPR ( x∗ ) : = κ∗∗ − k ⊤ f∗ ( Kff + σ 2In ) −1kf∗ , ( 1 ) where kf∗ is a vector of κ ( x∗ , xi ) as the i th element for all xi ∈ X , Kff is an n × n matrix with κ ( xi , xj ) as the ij th entry , and κ∗∗ = κ ( x∗ , x∗ ) . 2.1 Variational Expectation–Maximization for GPs with Non-Conjugate Likelihoods . For non-Gaussian likelihoods , the posterior and predictive distributions are no longer Gaussian , and we need to resort to approximate inference methods . Variational inference is a popular choice because it allows for fast posterior approximation and hyperparameter learning via stochastic training [ 35 , 10 ] . Denoting kernel hyperparameters by θ and the corresponding GP prior by pθ ( f ) , the posterior distribution can be written as pθ ( f |y ) = pθ ( f ) p ( y | f ) /pθ ( y ) , where pθ ( y ) = ∫ pθ ( f , y ) df is the marginal likelihood of the observations . We seek to approximate pθ ( f |y ) ≈ qf ( f ) by a Gaussian distribution whose parameters can be obtained by optimizing the following evidence lower bound ( ELBO ) to the log-marginal likelihood , log pθ ( y ) ≥ Lq ( qf , θ ) = ∑n i=1 Eqf ( fi ) [ log p ( yi | fi ) ] −DKL [ qf ( f ) ‖ pθ ( f ) ] . ( 2 ) For the variational approximation , it is a standard practice to choose the mean-covariance parameterization , denoted by ξ = ( m , S ) . It is also common to use the Cholesky factor L instead [ 3 ] since it is uniquely determined for a covariance matrix . The multivariate normal distribution is part of the exponential family [ 37 ] , i.e . it ’ s probability density function take the form p ( x ) = exp ( η⊤T ( x ) − a ( η ) ) , with natural parameters η = ( S−1m , −S−1/2 ) and sufficient statistics T ( x ) = [ x , xx⊤ ] . This natural parameterization is also a common choice , along with the associated expectation parameterization µ = Eq [ T ( x ) ] = ( m , S+mm ⊤ ) . A final choice of the parameterization , called the whitened parameterization [ 36 ] , uses a variable v ∼ N ( v ; mv , Sv ) along with the transformation f = Lv , to parameterize m = Lmv and S = LSvL ⊤ . In the following , we will consider the ELBO with several parameterizations and , to make the notation clearer , we will indicate the parameterization used with a subscript with L. For example , we may have Lξ ( ξ , θ ) = Lη ( η , θ ) = Lµ ( µ , θ ) , which are all clearly equal due to a unique mapping between the parameterizations ; see [ 25 ] for more details on the maps . The ELBO can be optimized by using a variational expectation–maximization ( VEM ) procedure , where we alternate between optimizing variational parameters , say ξ , and hyperparameters θ , E-step : ξ∗t ← argmaxξ Lξ ( ξ , θt ) , M-step : θt+1 ← argmaxθ Lξ ( ξ ∗ t , θ ) , ( 3 ) where t denotes the iterations , and we have explicitly written the dependence of optimal parameter ξ∗ ( θt ) as a function of the old parameter θt . Both and E and M-steps can be carried out with gradient descent , for example , using an iteration of the form ξ ( k+1 ) t ← ξ ( k ) t + ρk∇ξLξ ( ξ ( k ) t , θt ) for E-step , which would ultimately converge to ξ∗t . A similar iterative method can be used for the M-step . 2.2 Inference via Natural-Gradient Descent ( NGD ) . A popular strategy for the E-step is to use natural gradient descent where we replace the gradient by the one preconditioned using the Fisher information matrix F ( ξ ) of qf ( f ) . We denote natural gradients by ∇̃ξLξ ( ξ , θ ) = F ( ξ ) −1∇ξLξ ( ξ , θ ) , to get the following update , ξ ( k+1 ) t ← ξ ( k ) t + ρk∇̃ξLξ ( ξ ( k ) t , θt ) . ( 4 ) Such updates can converge faster than gradient descent [ 33 , 17 , 18 ] , and at times are less sensitive to the choice of the learning rate ρk due to the scaling with the Fisher information matrix . The implementation simplifies greatly when using natural parameterizations , η ( k+1 ) t ← η ( k ) t + ρk∇µLµ ( µ ( k ) t , θt ) , ( 5 ) because ∇̃ηLη ( η , θ ) = ∇µLµ ( µ , θ ) , that is , the natural gradients with respect to η are in fact the gradients with respect to the expectation parameter µ [ 18 ] . In the remainder of the paper we will frequently use this property , and refer to the natural gradient with respect to η by the gradients with respect to expectation parameterization µ . The VEM procedure with NGD has recently become a popular choice for sparse variants of GPs , which we explain next .

Summary: The paper revisits a known, but seemingly forgotten dual parametrization for multivariate Gaussian distributions that naturally lends itself to variational inference in Gaussian process models. This dual parametrization was introduced in previous works for variational Gaussian inference using a Lagrangian relaxation of the ELBO loss. Similarly to commonly used parametrizations, the dual parametrization can be combined with natural gradients based optimization that allows for faster training of the variational distribution by exploiting the information geometry of distribution space. However, the required natural gradients are considerably cheaper to compute in this setting due to only requiring first-order gradients. This parametrization is then incorporated into a sparse variational inference scheme for GPs. One of the computational difficulties when training the the SVGP lies with the coupled optimization of the variational distribution and the prior hyperparameters, for which several heuristics have been introduced in the community. The experiments compare the proposed parametrization for SVGP with the commonly used whitened one. They demonstrate that for small and mid-sized datasets, gradient descent along the trajectory of optimal variational distributions results in more stable learning of the hyperparameters. For larger datasets, the coupled stochastic optimization of the two (with only partial optimization of the variational distribution) leads the dual formulation to equivalent or slightly better optima measured in terms of predictive density, but it is considerably faster than the baseline due to cheaper natural gradients.