7 of the Most Used Regression Algorithms and How to Choose the Right One
By: Dominik Polzer
Tags: Editors Pick, Random Forest Regressor, Getting Started, Support Vector Regression, Linear Regression

Gaussian process regression

The Gaussian Process captures the typical behavior of a system on the basis of observations of a system and delivers as a result a probability distribution of possible interpolation functions for the problem at hand.

The Gaussian Process Regression makes use of the Bayes’ theorem in the following, which is why it should be briefly explained in advance.

In general, the Bayes’ theorem is defined as follows:

It allows the inference from known values to unknown values. A often used application example is the disease detection. In the case of rapid tests, for example, one is interested in how high the actual probability is, that a positive tested person actual has the disease. [Fah16]

In the following, we will apply this principle to the Gaussian Process.

The Gaussian Process is defined by the expected value for each random variable, the mean function m(x) and a covariance function k(x,x´) .

The mean function m(x) reflects the priori function for the problem at hand and is based on known trends or biases in the data. If the expected value (mean function) is constant 0, it is called a centered Gaussian process.

The covariance function k(x, x´) , also called ‘kernel’, describes the covariance of the random variables x and x′ . These functions are analytically defined.

The kernel defines the shape and the course of the model functions and is used to describe, for example, abstract properties such as smoothness, roughness and noise. More kernels can be combined by certain computational rules to emulate systems with superimposed properties. [Ebd08][Kuß06] [Ras06][Vaf17]

In the following three of the most used kernels will be presented:

Squared Exponential Kernel

A popular Kernel is the Squared Exponential Kernel (Radial Basis Function) and has established itself as the ‘standard kernel’ for the Gaussian Process and the Support Vector Machine. [Sci18l]

The following figures show an example for an A-priori-Gaussian process p(f) through the mean function m(x) (black line) and the confidence interval (gray background). Generally, the confidence interval indicates the range, given an infinite repetition of a random experiment, with some probability, the true location of the parameter lies [Fah16][Enc18]. In this case, the boundaries of the confidence interval are defined by the standard deviation σ.

The colored curves represent some random functions of the Gaussian Process. The example curves serve only to abstract the form of the possible output functions. In principle, an infinite number of these curves could be created.

A-priori Gaussian-Prozess using a Squared Exponential Kernel — Image by the author(inspired by [Sci18n][Duv14])

The kernel has only two hyperparameters:

l (length_scale) decribes the characteristic length scale of the covariance function. The length_scale influences the length of the ‘waves’ of the Gaussian functions.

decribes the characteristic length scale of the covariance function. The influences the length of the ‘waves’ of the Gaussian functions. The variance σ² defines the average distance of the function from its mean. The value should be chosen high for functions that cover a large range on the y-axis. [Ebd08]

The following figure shows the effects of the hyperparameters on the A-Priori-Gaussian-Process and its functions.

Squared Exponential Kernel: Influence of Hyperparameters — Image by the author

Rational Quadratic Kernel

The Rational Quadratic Kernel can be seen as a combination of several Squared Exponential Kernels with different length_scale settings (l). The parameter α determines the relative weighting of the 'large-scale' and 'small-scale' functions. When α approaches infinity, the Rational Quadratic Kernel is identical to the Squared Exponential Kernel. [Duv14][Sci18k][Mur12]

A-priori Gaussian-Prozess using Rational Quadratic Kernel — Image by the author (inspired by [Sci18n][Duv14])

Periodic Kernel

The Periodic Kernel allows functions to repeat themselves. The period p describes the distance between the repetitions of the function. The use of the ‘lenghtscale' parameter (l) is as mentioned earlier. [Sci18j]

A-priori Gaussian-Prozess using Periodic Kernel — Image by the author (inspired by [Sci18n][Duv14])

The kernel funnction and mean function together describe the A-priori-Gaussian-Process. With the help of some measured values, a A-posteriori-Gaussian-Process can be defined, which takes into account all available information about the problem. More precisely, no single solution results, but all possible functions of the interpolations, which are weighted with different probabilities. In the case of a regression task, specifically, the solution (function) with the highest probability is crucial. [Ras06][Wik18a][Wik18a]

For regression, typically a dataset with values of the independent variable X ∈ R and associated values of the dependent variable f ∈ R is given and one wants to predict output values f∗ for new values X∗. [Vaf17]

For the simplest case, a process without noise, the multidimensional Gaussian distribution is defined as follows:

The covariance matrix can be divided into four parts. The covariance within the unknown values K_X∗X∗ , the covariance between the unknown and known K_X∗X values, and the covariance within the known values K_XX.

Since f is completely known, substituting the probability density into Bayes’ theorem yields the a-posterior-Gaussian-distribution.

A detailed derivation is given by Rasmussen in his book ‘Gaussian Processes for Machine Learning’. [Ras06, p.8 ff.]

From priori to posteriori Gaussian Process: Explained with a simple example

In practice, a number of other kernels are used, including combinations of several kernel functions. The constant kernel for example is usually used in conjunction with others. Using this kernel without the combination with other kernels, usually makes no sense, because only constant correlations can be modeled. Nevertheless, in the following the constant kernel is used to explain and illustrate the Gaussian-process regression in a simple way.

The figure below shows the a-priori-Gaussian-Process with a variance of one. By defining the constant kernel as the covariance function, all sample functions show a parallel line to the x-axis.

Representation of the Gaussian process with the so-called constant kernel and one supporting data point — Image by the author

Since no statement is made in advance about a possible noise of the measured data, the process assumes that the given measurement point is a part of the true function. This limits the number of possible function equations to the straight line that passes directly through the point. Since the constant kernel only allows horizontal lines, the number of possible lines in this simple case narrows down to exactly one possible function. Thus the covariance of the a-posteriori-Gaussian process is zero.

With the RBF Kernel , arbitrary processes can be mapped, but this time the result is not a single straight line as A-posteriori-Gaussian, but a multitude of functions. The function with the highest probability is the mean function of the A-posteriori-Gaussian-Process. The following figure shows the A-posteriori-Gaussian-Process and the used measurement points.

Posterior Gaussian Process with Squared Exponential Kernel and used data points — Image by the author

To implement the Gaussian Process in Python, the A-priori-Gaussian-Process must be defined in advance. The mean function m(x) is usually assumed to be constant and zero. By setting a parameter normalize_y = True , the process uses the mean of the dataset values as the constant expected value function. The choice of the covariance function is made by choosing the kernel. [Sci18m]

Gaussian Process Regression in Scikit-learn

The following source code describes how to implement the Gaussian Process Regression with scikit learn and the RBF Kernel used as covariance function. The first optimization process starts from the pre-set values ( length_scale and variance ) of the kernel. By the parameter alpha an assumption can be made in advance about the strength of the noise of the training data.

Optimization process: Hyperparameter erstimation using maximum-likelihood method

The hyperparameters are optimized during the fitting of the model by maximizing the log-marginal likelihood (LML). Maximum Likelihood Estimation (MLE) is a method for determining the parameters of a statistical model. While the Regression methods already presented, such as Linear Regression, aim to minimize the Mean Square Error, the Gaussian-Process Regression tries to maximize the likelihood function. In other words, the parameters of the model are selected in such a way that the observed data appear most plausible according to their distribution.

In general, the probability function f of a random variable X is defined as:

This distribution is assumed to depend on a parameter ϑ. Given observed data, the probability can be considered as a function of ϑ :

Maximum likelihood estimators aim to maximize this function. Maxima are usually identified by differentiating the function and then setting it equal to zero. Since the log likelihood function has its maximum at the same point as the likelihood function, but is easier to calculate, it is usually used. [Goo16, p.128]

One speaks of a normal or Gaussian distribution if a random variable X has the following probability density [Fah16, p.83]:

The maximum-likelihood method will be explained in the following using a simple one-dimensional example. The figure below shows the dataset. All three plotted probability distributions reflect the distribution of the data. With maximum likelihood, one is interested in the distribution that is most likely.

The normal distribution depending on the parameters expected value and variance — Image by the author (inspired by [BB18])

The goal is to define the parameters σ² and µ in such a way that the probability is maximized over all data points considered. For example, given three data points with x-values 9, 10 and 13, similar to the data points in the figure, the joint probability is calculated from the individual probabilities as follows:

This function must be maximized. The mean value then corresponds to the x-value that is most likely to occur.

The following figure shows an example model for the Gaussian Process Regression.