shogun-toolbox/shogun — pull request #707 by puffin444 (2012-08-13 08:20:14+00:00)
Title: Final Features

Hi all,

Here are finally the last features of the project.
I am so sorry this is late; I wrote this up last
week, but I was bogged down by awful numerical
bugs.

In this patch I:

1. Fixed all my doxygen warnings
2. Added the scale feature to the FITC inference
method.
3. Added Laplace Infererence Approximation
4. Added Student's T likelihood model
5. Added a corresponding example of the laplacian
Approximation using the Student's T likelihood model.
6. Put all of this into the modular framework.

Here are a few things:

1. In order to do FITC scaling, I sort of repeated my
code in the gradient calculation function in CFITCInferenceMethod.
In order to cut down on redundancy, should I try to add a private
function that handles both the kernel derivatives and scaling?

2. It doesn't seem that Eigen likes empty matrices, so in order
to let the inference method know that a parameter does not belong
to a likelihood function, the likelihood function returns a derivative
vector to the inference method full of infinity values. This way,
the inference method knows that the parameter does not belong to that
likelihood function. Is this a bad way to do things?

3. I had to add an implementation of the Brent Minimization
algorithm for laplacian inference. The source is right here:
http://people.sc.fsu.edu/~jburkardt/m_src/brent/brent.html

4. Because of the high computation complexity of these features,
I have closely adapted the MATLAB code from GPML for the Laplacian
Method and the Student's T likelihood function. I acknowledge this
in the source code files. Is this okay?

5. When calculating the Newton iterations in the laplacian method, some
likelihood functions create a negative W noise matrix. GPML's method of handling
this case seemed to be unstable on my system, so instead I tried their
suggested alternative by using the degrees of freedom. Of course, not all likelihood
functions have degrees of freedom as a parameter. Does it make sense to set
this to 1 for such likelihood functions, or does this sound like a stupid idea? (be honest)

7. I had to adapt a function that calculates the derivative of the gamma function for
the student's t likelihood distribution. Should this be integrated into shogun's math library?

8. I had some stability issues calculating the student t likelihood derivatives directly, so I instead
calculated them in log space like GPML. In order to convert back to parameter space, does it make
mathematical sense to divide by the variable, i.e if the derivative is dy/dlog(x), can I simply multiply
this by dlog(x)/dx to obtain dy/dx? Or am I having a stupid math day?

Thanks,

Jacob