scikit-learn/scikit-learn — pull request #1333 by eickenberg (2012-11-05 16:56:29+00:00)
Title: WIP: Multi target ridge with individual penalties

Hi everybody,

here is an augmented version of linear_model.ridge.ridge_regression. It adds the solvers "svd" and "eigen". Using one decomposition of the design matrix, these solvers handle multiple targets y and individual penalties for each target.
Penalties are passed using the alpha-argument using an array or an iterable with length corresponding to the number of targets.

This is a first attempt at rendering this idea sklearn compatible and I am not sure it is yet, so any feedback is very welcome!

As of now, what is still missing is a certain number of decisions in the beginning of the function, i.e. if multiple penalties for multiple targets are passed, then the right solver should be chosen automatically, I guess.

Both SVD and eigen are provided, because I have been having to deal with ill-conditioned matrices latelely which provoke a "SVD did not converge" from scipy due to some infinitesimally negative eigenvalues of the usually positive semidefinite matrix np.dot(X, X.T). Eigen seems to be better adapted here.

I added these solvers to the test_ridge.test_ridge() and they pass, but this does not test for multiple penalties yet.

So TODO:
- boiler plate decision code at the beginning of ridge_regression
- tests for the new functionality

As a matter of fact I also plan to add a general fold-wise cross-validation scheme to RidgeCV which should benefit from the fact that one can test several penalties at once given a decomposition of the design matrix. For this reason, I have rendered the functionality of what I added to ridge_regression more general:

If passing individual penalties to ridge_regression using an array alpha, then we impose

y.shape[1] = alpha.shape[-1]

All other potential dimensions of alpha can be used to try several different penalties per target.

I know this feature smells strongly of YAGNI but I have been using it for an efficient grid search procedure in cross-validation.

One last question: Any suggestions on how to introduce a caching option for the decomposition using joblib.Memory? Options would be 1) Pass your own SVD, having decorated it with Memory, or 2) pass your own Memory which defaults to the transparent mem = Memory(cachedir=None).

Thanks for your time!
Michael