scikit-learn/scikit-learn — pull request #2930 by jmetzen (2014-03-03 14:49:05+00:00)
Title: [MRG] Enhanced correlation models and noise estimation for Gaussian Process

Support in Gaussian-Process regression for enhanced correlation models and learning the noise magnitude (the nugget) from training data.

Correlation models have been extended as follows:
* Matern correlation models for nu=1.5 and nu=2.5 have been added (see https://en.wikipedia.org/wiki/Mat%C3%A9rn_covariance_function). An example script showing the potential benefit of the Matern correlation model compared to squared-exponential and absolute-exponential was added under examples/gaussian_process/plot_matern_kernel.py (see attached image)
* squared_exponential, absolute_exponential and Matern correlation models support factor analysis distance. This can be seen as an extension of learning dimension-specific length scales in which also correlations of feature dimensions can be taken into account. See Rasmussen and Williams 2006, p107 for details. An example script showing the potential benefit of this extension was added under examples/gaussian_process/plot_gp_learning_curve.py (see attached image). This feature required that correlation modes get passed the componentwise differences rather than the componentwise distances (their absolute value).

Learning the noise (the "nugget effect") by GaussianProcess is now supported by setting the parameter learn_nugget to True. This allows to learn a homoscedastic noise model, i.e., it assumes that the noise has globally the same magnitude. The script examples/gaussian_process/plot_gp_regression.py was modified accordingly, i.e., it learns the noise magnitude and does not rely on specifying it externaly.

Furthermore, a typo in gp_diabetes_dataset.py was fixed and a not yet merged bugfix (#2867 and #2798) is included.

To be discussed:
* The factor analysis distance has hyperparameters that can can take on arbitrary real values (not just positive ones). Since sklearn enforces the hyperparameters theta to be positive, this is internally handled by taking the log of the corresponding components of theta. Are there better ideas?
* Learning the noise (the nugget) is internally handled by appending it to the vector theta in _arg_max_reduced_likelihood_function(). This was the way which required the least changes in the current implementation but is not necessarily the best way. Are their any opinions on that?

![plot_gp_learning_curve](https://f.cloud.github.com/assets/1116263/2310367/a7499dde-a2e2-11e3-80f5-6e98a23f0ee8.png)
![plot_matern_kernel](https://f.cloud.github.com/assets/1116263/2310372/adf004fc-a2e2-11e3-8d25-85e69e09fd11.png)