scikit-learn/scikit-learn — pull request #2327 by fabianp (2013-07-29 23:46:37+00:00)
Title: Implementation of logistic_regression_path and LogisticRegressionCV

This code implements a function logistic_regression_path and an object LogisticRegressionCV

This is mostly the code from @GaelVaroquaux logistic_cv branch. My work is only some minor cleanup some added tests and benchmarking.

## Rationale

We currently use LIBLINEAR for LogisticRegression and while it is very fast on most problems it cannot be warm-restarted, which means that fitting a LogisticRegression on a grid of parameters takes as much time as fitting it independently on each parameter separately. In this pull request we implement the code to perform an efficient fit of logistic regression models across a sequence of regularization parameters by using the solution to the previous model as initialization point for the current optimization problem. We also get (almost for free) an efficient implementation of a cross-validated Logistic Regression estimator in LogisticRegressionCV

## Code

The important functions are ``logistic_regression_path`` and ``LogisticRegressionCV``, both in ``linear_models/logistic.py``. Our benchmarks have shown that the most competitive method is either lbfgs or the newton-cg solver adapted from the SciPy codebase, depending on the problem. The changes made to the newton-cg code are essentially to allow the function to minimize to provide the gradient and hessian in the same call, thus being able to reuse several computations. These changes are extremely important for this model but unfortunately these changes are incompatible with the SciPy API and extending the SciPy API to support our changes is not straightforward. That is why I've added the solver to sklearn.utils.optimize.

The available solvers are (I've tried to use as much the vocabulary from scipy.optimize):

* **lbfgs**: this uses scipy.optimize.fmin_lbfgs_b

* **newton-cg**: this uses our modificed version of scipy's newton-cg implementation

* **liblinear**: this uses the LIBLINEAR implementation, which in turn is a trust-region newton type algorithm.

The solvers make use of the private functions ``_logistic*`` defined in ``logistic.py``.

## Benchmarks

I have run benchmarks on real and synthetic datasets.

Each vertical line corresponds to the convergence of the method for a particular parameter and each axis is done with a different method. The important quantity is the timing for the last line, that is, when the algorithm finishes to optimize all models in the path (i.e. total time).

### Real datasets

This if from the haxby dataset from https://github.com/nilearn/nilearn. 216 samples x 6222 features. I'll upload this data somewhere as soon as I have decent wifi connection. The notebook to run the benchmarks can be found [here](http://nbviewer.ipython.org/url/fa.bianp.net/tmp/2013/logistic_path_bench_PR.ipynb).

![image](https://f.cloud.github.com/assets/277639/875580/365ac6d4-f8a6-11e2-824c-f0628d95cc58.png)

### Synthetic dataset

From a linear model with gaussian noise (see notebook), 10K samples x 100 features.

![image](https://f.cloud.github.com/assets/277639/875643/fd2b3d38-f8a7-11e2-877c-b492fc18fa1b.png)

## TODO

I think the some of the private functions in ``logistic.py`` can be simplified. In particular, it is my intuition that we could squash some of the functions ``_logistic*`` and ``_logistic_*_intercept`` without sacrificing a significant amount of performance