Advances in Approximate Bayesian Inference for Gaussian Process Models

Aalto University, P.O. Box 11000, FI-00076 Aalto www.aalto. fi Author Jaakko Riihimäki Name of the doctoral dissertation Advances in Approximate Bayesian Inference for Gaussian Process Models Publisher School of Science Unit Department of Biomedical Engineering and Computational Science Series Aalto University publication series DOCTORAL DISSERTATIONS 144/2013 Field of research Computational Engineering Manuscript submitted 11 June 2013 Date of the defence 11 October 2013 Permission to publish granted (date) 16 August 2013 Language English Monograph Article dissertation (summary + original articles) Abstract Gaussian processes (GPs) provide a flexible approach to construct probabilistic models for Bayesian data analysis. In the Bayesian approach, GPs are used to specify prior assumptions on the latent function values that describe the underlying relationships between the explanatory variables and the associated target variables. These prior assumptions are combined with information from the observations using Bayes' rule. The obtained result is the posterior distribution that represents the uncertainty about the latent function values of interest, conditioned on the observations and the model assumptions. A challenge with the Bayesian approach is that exact inference is analytically intractable to calculate for most GP models used in practice. Therefore, approximate methods are needed in order to evaluate the posterior distribution and to make predictions for new observations.Gaussian processes (GPs) provide a flexible approach to construct probabilistic models for Bayesian data analysis. In the Bayesian approach, GPs are used to specify prior assumptions on the latent function values that describe the underlying relationships between the explanatory variables and the associated target variables. These prior assumptions are combined with information from the observations using Bayes' rule. The obtained result is the posterior distribution that represents the uncertainty about the latent function values of interest, conditioned on the observations and the model assumptions. A challenge with the Bayesian approach is that exact inference is analytically intractable to calculate for most GP models used in practice. Therefore, approximate methods are needed in order to evaluate the posterior distribution and to make predictions for new observations. This thesis develops methods for approximate Bayesian inference in various modelling problems involving GP models. The focus is on efficient ways to form Gaussian posterior approximations based on Laplace's method or expectation propagation (EP). The inference for the studied GP models is challenging in two aspects. Firstly, observation models are generalized in the way that the probability distribution for each observation can depend on multiple values of the latent function instead of only one value, or on the derivative values of the latent function. Secondly, instead of one prior process, the models can have multiple uncorrelated prior processes that are coupled through the observation model. This thesis presents improvements to approximate Bayesian inference for GP models in density estimation, survival analysis, and multiclass classification. We describe Laplace's method for a logistic GP model and for a Cox-type survival model constructed from GP priors to speed up the inference. We develop a novel nested EP algorithm for multinomial probit GP classification that does not require numerical quadratures and scales linearly in the number of classes. In addition, we extend the existing methodology proposed for regression and binary classification by introducing monotonicity information into a GP model with EP. We demonstrate the practical accuracy of the described methods with several experiments and apply them to real-life modelling problems.