scikit-learn/scikit-learn — pull request #2340 by dashesy (2013-08-03 00:25:41+00:00)
Title: [WIP] Add Gaussian Process Classification (GPC) with Laplace approximation

This is almost entirely based on "Gaussian Processes for Machine Learning" by Rassmussen and Williams.

Please let me know where I should add test methods, and anything else I need to do (not sure but cannot subscribe to the mailing list to ask my questions there).
I can add real benchmarks as soon as I get some feedback, and hopefully some pointers.

Code
=====
I followed the newton formulation described in the text, the method directly computes the inverse Hessian, so optimize.fmin_ncg was not used, it has the advantage that kernel does not need to be inverted, and avoids numerical instability (as described in the book). I have an implementation that does use fmin_ncg but requires the invert, it can be added as a separate method to the class.

Comparison
============
It seems Laplace approximation to find the posterior distribution is not well-received in the literature, but it should be a good reference.
since the Expectation Propagation (EP) method described there is for binary classification only, I just implemented the Laplace approximation method for now, if there is any good reference for EP or other methods please let me know. Another option is to use EP in a ovR classification.
Variational methods also mentioned in the book, and seem to be easier to implement than EP for multiple classes.
If integrating with PyMC is an option, an MCMC approach would be a nice addition too.

Rationale
=======
Gaussian Process Classification (GPC) is a kernel-based non-parametric method. One main advantage is flexibility and noise tolerance. It is a natural generalization of linear logistic regression, similar to the transition from linear regression to GP regression. The training learns a set of functions.
It is also the starting point to some powerful dimensionality reduction (DR) techniques that are based on GP-LVM, and learn the manifold of high-dimensional data with few number of samples.

TODO
=====
I wanted to get some feedback before I continue, but here are my TODO list:
1- Hyper parameter theta can be optimized in a straight forward approach similar to GPR
2- Add numerical integration for prediction instead of Monte Carlo (MC) sampling (I wanted to follow the text book)
3- Add a variational method or EP method if I find good references within my level of math

My plan is to continue by implementing latent variable model (GP-LVM) and make it supervised (SGP-LVM). It will be based on these papers:

"Supervised Gussian Process LVM for Dimensionality Reduction", Xinbo Gao
"Supervised Latent Linear Gaussian Process LVM Based Classification", Hou, Feng, Zou
"Supervised Latent Linear Gaussian Process LVM for Dimensionality Reduction", Jiang, Gao, Wong, Zheng
"Discriminative Gaussian Process LVM for Classification", Urtasun and Darrell