Repository: mne-tools/mne-python

Issue:
ENH: better solver for rERP, encoder models redux
Ridge and OLS as implemented in sklearn are much faster than the current solver, for some reason. (The reason is that obviously, something I hack together will always be worse than sklearn.)
They also are better at optionally dealing with sparse input.
I'd like to add them, and make them the go-to with the current one as a fallback.

(This is another incremental step of expanding rERP/linear_regression_raw into an encoder model.)

I'll go about implementing the first one soon.

Q: what's the most economic way of checking if a scipy coc sparse array is truly sparse, or alternatively, what's a good way of checking if it should be treated as sparse by the solver, or converted to dense and solved by a normal solver?



Discussion:
> Q: what's the most economic way of checking if a scipy coc sparse array is truly sparse

Depends on what you mean by that, but this would get you some of the way maybe

http://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.sparse.csr_matrix.nnz.html

E.g. you can currently use this for the solver param and it works better:

```
def solver(X, y):
    from sklearn.linear_model import Ridge
    return Ridge().fit(X, y.T).coef_
```

Also works with RidgeCV. Not really with Lasso though ..

Thanks Eric, that's probably it. Unless anybody has a rule of thumb regarding up to what sparsity/density it makes sense to leave X as sparse/use a sparse solver.

I think you should always leave it sparse and leave it up to the user to do the right thing in terms of converting it or not

Sadly, this is not possible for the user-facing rERP function, because X exists only inside the function. The user doesn't supply X, they supply the information to construct X, and get the evokeds.

so the rERP function decides whether to construct a sparse matrix or dense array?

Due to the way we've settled on handling it internally, it always starts out as sparse (see https://github.com/mne-tools/mne-python/blob/master/mne/stats/regression.py#L363), and right now, the solver always converts it to dense (see https://github.com/mne-tools/mne-python/blob/master/mne/stats/regression.py#L254).

The part where Denis and I settled on using sparse.dia_matrix is handled differently by Chris' approach, by XDawn, by the original implementation, and was handled differently by the initial version of linear_regression_raw. The current version is the only one of these that's sparse, and that was because we saw major performance improvements from that.

> Unless anybody has a rule of thumb regarding up to what sparsity/density it makes sense to leave X as sparse/use a sparse solver.

Ahh, that's what you mean by this. I don't know any rule of thumb, but you could do some benchmarking to try to see when approximately it switches.

> The current version is the only one of these that's sparse, and that was because we saw major performance improvements from that.

To start, though, I'd just try to get it working and leave such optimization for later (with notes in the code to come back to it). Trying to optimize with such branching while also expanding / unifying funtionality can make things more difficult. To do it properly I'd verify that the dense and sparse paths give equivalent results (meaning unit tests) and then do benchmarking, etc.

Actually, I just realized the user can indeed decide on converting to dense by plugging in a solver that converts to dense vs. one that doesn't. But I always like an 'auto' option ... I'll play around with it a bit.

E.g. the user can do

```
def solver_sparse(X, y):
    from sklearn.linear_model import Ridge
    return Ridge().fit(X, y.T).coef_
```

and

```
def solver_dense(X, y):
    from sklearn.linear_model import Ridge
    return Ridge().fit(X.toarray(), y.T).coef_
```

and then

```
evokeds = linear_regression_raw(raw, events, event_id, solver=solver_dense, ...)
```


Resolving patch:
diff --git a/mne/stats/regression.py b/mne/stats/regression.py
--- a/mne/stats/regression.py
+++ b/mne/stats/regression.py
@@ -16,7 +16,7 @@
 from ..source_estimate import SourceEstimate
 from ..epochs import _BaseEpochs
 from ..evoked import Evoked, EvokedArray
-from ..utils import logger, _reject_data_segments, _get_fast_dot, warn
+from ..utils import logger, _reject_data_segments, warn
 from ..io.pick import pick_types, pick_info
 from ..fixes import in1d
 
@@ -155,7 +155,7 @@ def _fit_lm(data, design_matrix, names):
 
 def linear_regression_raw(raw, events, event_id=None, tmin=-.1, tmax=1,
                           covariates=None, reject=None, flat=None, tstep=1.,
-                          decim=1, picks=None, solver='pinv'):
+                          decim=1, picks=None, solver='cholesky'):
     """Estimate regression-based evoked potentials/fields by linear modelling
 
     This models the full M/EEG time course, including correction for
@@ -228,8 +228,8 @@ def linear_regression_raw(raw, events, event_id=None, tmin=-.1, tmax=1,
     solver : str | function
         Either a function which takes as its inputs the sparse predictor
         matrix X and the observation matrix Y, and returns the coefficient
-        matrix b; or a string (for now, only 'pinv'), in which case the
-        solver used is `dot(scipy.linalg.pinv(dot(X.T, X)), dot(X.T, Y.T)).T`.
+        matrix b; or a string. If str, must be `cholesky`, in which case the
+        solver used is ``linalg.solve(dot(X.T, X), dot(X.T, y))``.
 
     Returns
     -------
@@ -246,13 +246,12 @@ def linear_regression_raw(raw, events, event_id=None, tmin=-.1, tmax=1,
     """
 
     if isinstance(solver, string_types):
-        if solver == 'pinv':
-            fast_dot = _get_fast_dot()
+        if solver == 'cholesky':
+            def solver(X, y):
+                a = (X.T * X).toarray()  # dot product of sparse matrices
+                return linalg.solve(a, X.T * y.T, sym_pos=True,
+                                    overwrite_a=True, overwrite_b=True).T
 
-            # inv is slightly (~10%) faster, but pinv seemingly more stable
-            def solver(X, Y):
-                return fast_dot(linalg.pinv(X.T.dot(X).todense()),
-                                X.T.dot(Y.T)).T
         else:
             raise ValueError("No such solver: {0}".format(solver))
 


Test patch: