Query: gaussian process hyperparameters

title: Task Agnostic Amortized Inference of Gaussian Process Hyperparameters
abstract: Gaussian processes (GPs) are flexible priors for modeling functions. However, their success depends on the kernel accurately reflecting the properties of the data. One of the appeals of the GP framework is that the marginal likelihood of the kernel hyperparameters is often available in closed form, enabling optimization and sampling procedures to fit these hyperparameters to data. Unfortunately, point wise evaluation of the marginal likelihood is expensive due to the need to solve a linear system; searching or sampling the space of hyperparameters thus often dominates the practical cost of using GPs. We introduce an approach to the identification of kernel hyperparameters in GP regression and related problems that sidesteps the need for costly marginal likelihoods. Our strategy is to "amortize" inference over hyperparameters by training a single neural network, which consumes a set of regression data and produces an estimate of the kernel function, useful across different tasks. To accommodate the varying dimension and cardinality of different regression problems, we use a hierarchical self attention based neural network that produces estimates of the hyperparameters which are invariant to the order of the input data points and data dimensions. We show that a single neural model trained on synthetic data is able to generalize directly to several different unseen real world GP use cases. Our experiments demonstrate that the estimated hyperparameters are comparable in quality to those from the conventional model selection procedures, while being much faster to obtain, significantly accelerating GP regression and its related applications such as Bayesian optimization and Bayesian quadrature. The code and pre trained model are available at https: /github.com/PrincetonLIPS/AHGP.

title: On the Choice of Optimisation Scheme for Gaussian Process Hyperparameters in SHM Problems
abstract: Within Structural Health Monitoring a number of problems exist which require estimation and prediction of quantities such as load, strain, or acceleration. This can be achieved through Gaussian Process (GP) regression models, which provide a powerful framework for Bayesian machine learning. The ability of GP models to automatically return confidence intervals and not needing to specify a topology, as with Artificial Neural Networks, makes them an appealing choice in engineering problems where quantifying uncertainty is important to the model application. Although GP models are nonparametric, at the heart of the model is the optimisation of the marginal likelihood of the data with respect to the hyperparameters which govern the covariance function. Within many software packages that implement GP models, this optimisation is achieved via a conjugate gradient descent, since the derivatives of the marginal likelihood are cheap to compute once the function value is obtained. This paper investigates the effect of choice of hyperparameter optimisation scheme on the quality of fit of the model and the training time to determine if a conjugate gradient optimisation is sufficient to identify the covariance function hyperparameters. A number of optimisation schemes are tested on synthetic data, and also on data from in flight monitoring of aircraft dynamics. Specifically the problem of strain prediction for fatigue monitoring of a Tucano TMk1 trainer aircraft is considered. The effect of input data dimensionality on the ability of the optimisation schemes to find hyperparameters which suitably describe the data is also considered. This paper aims to begin a discussion into when the cost of more complex optimisation schemes is justified for Gaussian process hyperparameter optimisation.

title: Marginalizing Gaussian process hyperparameters using sequential Monte Carlo
abstract: Gaussian process regression is a popular method for non parametric probabilistic modeling of functions. The Gaussian process prior is characterized by so called hyperparameters, which often have a large influence on the posterior model and can be difficult to tune. This work provides a method for numerical marginalization of the hyperparameters, relying on the rigorous framework of sequential Monte Carlo. Our method is well suited for online problems, and we demonstrate its ability to handle real world problems with several dimensions and compare it to other marginalization methods. We also conclude that our proposed method is a competitive alternative to the commonly used point estimates maximizing the likelihood, both in terms of computational load and its ability to handle multimodal posteriors.

title: Optimization of Gaussian process hyperparameters using Rprop
abstract: Gaussian processes are a powerful tool for non parametric re gression. Training can be realized by maximizing the likelihood of the data given the model. We show that Rprop, a fast and accurate gradient based optimization technique originally designed for neural network learning, can outperform more elaborate unconstrained optimization methods on real world data sets, where it is able to converge more quickly and reliably to the optimal solution.

title: How priors of initial hyperparameters affect Gaussian process regression models
abstract: The hyperparameters in Gaussian process regression (GPR) model with a specified kernel are often estimated from the data via the maximum marginal likelihood. Due to the non convexity of marginal likelihood with respect to the hyperparameters, the optimisation may not converge to the global maxima. A common approach to tackle this issue is to use multiple starting points randomly selected from a specific prior distribution. As a result the choice of prior distribution may play a vital role in the predictability of this approach. However, there exists little research in the literature to study the impact of the prior distributions on the hyperparameter estimation and the performance of GPR. In this paper, we provide the first empirical study on this problem using simulated and real data experiments. We consider different types of priors for the initial values of hyperparameters for some commonly used kernels and investigate the influence of the priors on the predictability of GPR models. The results reveal that, once a kernel is chosen, different priors for the initial hyperparameters have no significant impact on the performance of GPR prediction, despite that the estimates of the hyperparameters are very different to the true values in some cases.

title: Scalable Variational Bayesian Kernel Selection for Sparse Gaussian Process Regression
abstract: This paper presents a variational Bayesian kernel selection (VBKS) algorithm for sparse Gaussian process regression (SGPR) models. In contrast to existing GP kernel selection algorithms that aim to select only one kernel with the highest model evidence, our VBKS algorithm considers the kernel as a random variable and learns its belief from data such that the uncertainty of the kernel can be interpreted and exploited to avoid overconfident GP predictions. To achieve this, we represent the probabilistic kernel as an additional variational variable in a variational inference (VI) framework for SGPR models where its posterior belief is learned together with that of the other variational variables (i.e. inducing variables and kernel hyperparameters) In particular, we transform the discrete kernel belief into a continuous parametric distribution via reparameterization in order to apply VI. Though it is computationally challenging to jointly optimize a large number of hyperparameters due to many kernels being evaluated simultaneously by our VBKS algorithm, we show that the variational lower bound of the log marginal likelihood can be decomposed into an additive form such that each additive term depends only on a disjoint subset of the variational variables and can thus be optimized independently. Stochastic optimization is then used to maximize the variational lower bound by iteratively improving the variational approximation of the exact posterior belief via stochastic gradient ascent, which incurs constant time per iteration and hence scales to big data. We empirically evaluate the performance of our VBKS algorithm on synthetic and massive real world datasets.

title: The Gaussian process distribution of relaxation times: A machine learning tool for the analysis and prediction of electrochemical impedance spectroscopy data
abstract: Electrochemical impedance spectroscopy (EIS) is one of the most important techniques in electrochemistry. However, analyzing the EIS data is not a simple task. The distribution of relaxation times (DRT) method offers an elegant solution to this considerable challenge. In addition to that, the DRT method can be used to obtain the time characteristics of the electrochemical system under study. Though, deconvolving the DRT from the EIS data is an ill posed problem, which is particularly sensitive to experimental errors. Several well known approaches, including ridge regularization, can overcome this issue but they all require the use of ad hoc hyperparameters. Furthermore, most methods are not probabilistic and therefore do not provide any uncertainty on the estimated DRT. In this work, by assuming that the DRT is a Gaussian process (GP) it is not only possible to obtain the DRT mean and covariance from the EIS data but also to predict both the DRT and the imaginary part of the impedance at frequencies not previously measured. One important point to note is that, unlike other methods, the parameters that define the GP DRT model can be selected rationally by maximizing the experimental evidence. The GP DRT approach is tested using synthetic experiments for analyzing the consistency of the method and "real" experiments to gauge its performance for real data. The GP DRT model is shown to be able to manage considerable noise, overlapping timescales, truncated data, and inductive features. Considering the GP DRT framework developed and the results of the simulations, the GP DRT will likely inspire further studies on using a probabilistic approach to interpret EIS data.

title: Neural net induced Gaussian process regression for function approximation and PDE solution
abstract: Neural net induced Gaussian process (NNGP) regression inherits both the high expressivity of deep neural networks (deep NNs) as well as the uncertainty quantification property of Gaussian processes (GPs) We generalize the current NNGP to first include a larger number of hyperparameters and subsequently train the model by maximum likelihood estimation. Unlike previous works on NNGP that targeted classification, here we apply the generalized NNGP to function approximation and to solving partial differential equations (PDEs) Specifically, we develop an analytical iteration formula to compute the covariance function of GP induced by deep NN with an error function nonlinearity. We compare the performance of the generalized NNGP for function approximations and PDE solutions with those of GPs and fully connected NNs. We observe that for smooth functions the generalized NNGP can yield the same order of accuracy with GP, while both NNGP and GP outperform deep NN. For non smooth functions, the generalized NNGP is superior to GP and comparable or superior to deep NN.

title: Efficient Bayesian shape restricted function estimation with constrained Gaussian process priors
abstract: This article revisits the problem of Bayesian shape restricted inference in the light of a recently developed approximate Gaussian process that admits an equivalent formulation of the shape constraints in terms of the basis coefficients. We propose a strategy to efficiently sample from the resulting constrained posterior by absorbing a smooth relaxation of the constraint in the likelihood and using circulant embedding techniques to sample from the unconstrained modified prior We additionally pay careful attention to mitigate the computational complexity arising from updating hyperparameters within the covariance kernel of the Gaussian process. The developed algorithm is shown to be accurate and highly efficient in simulated and real data examples.

title: Estimation of municipal waste generation of Turkey using socio economic indicators by Bayesian optimization tuned Gaussian process regression
abstract: Accurate estimation of municipal solid waste (MSW) generation has become a crucial task in decision making processes for the MSW planning and management systems. In this study, the Gaussian process regression (GPR) model tuned by Bayesian optimization was used to forecast the MSW generation of Turkey. The Bayesian optimization method, which can efficiently optimize the hyperparameters of kernel functions in the machine learning algorithms, was applied to reduce the computation redundancy and enhance the estimation performance of the models. Four socio economic indicators such as population, gross domestic product per capita, inflation rate, and the unemployment rate were used as input variables. The performance of the Bayesian GPR (BGPR) model was compared with the multiple linear regression (MLR) and Bayesian support vector regression (BSVR) models. Different performance measures such as mean absolute deviation (MAD) root mean square error (RMSE) and coefficient of determination (R2) values were used to evaluate the performance of the models. The exponential GPR model tuned by Bayesian optimization showed superior performance with minimum MAD (0.0182) RMSE (0.0203) and high R2 (0.9914) values in the training phase and minimum MAD (0.0342) RMSE (0.0463) and high R2 (0.9841) values in the testing phase. The results of this study can help decision makers to be aware of social economic factors associated with waste management and ensure optimal usage of their resources in future planning.