Repository: mne-tools/mne-python

Issue:
BUG: sklearn cross_val_predict changed, now incompatible with SlidingEstimator with predict proba
sklearn `cross_val_predict` has recently changed, and is now incompatible with SlidingEstimator with classification estimators, as it requires `n_classes`

```python
import numpy as np
from mne.decoding import SlidingEstimator
from sklearn.linear_model import LogisticRegression, LinearRegression
from sklearn.model_selection import cross_val_predict
X = np.random.randn(100, 20, 10)
y = np.random.randint(0, 2, 100)

base_estimator = LinearRegression()
cross_val_predict(SlidingEstimator(base_estimator), X, y)  # ok

base_estimator = LogisticRegression()
cross_val_predict(SlidingEstimator(base_estimator), X, y)  # ok

base_estimator = LogisticRegression()
cross_val_predict(SlidingEstimator(base_estimator), X, y,
                             method='predict_proba')  # not ok
```

Three possible fixes:
~~1) Detect when `base_estimator ` is classifier, and add n_classes attribute to SlidingEstimator~~
2) make `mne.decoding.cross_val_multipredict`, similar to `mne.decoding.cross_val_multiscore`
3) add a 'method' argument to SlidingEstimator


Discussion:
don't we use cross_val_predict only in examples? if so why not passing
n_classes in our examples code?

I don't understand. 

It's more general issue of making SlidingEstimator compatible with sklearn, not just an example issue.
what do you want to do? what code breaks and should not?

? Please read above description, and possible fixes.
you need to add this:

https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/model_selection/_search.py#L553

to our metaestimators

Ok I tried, but it's more complicated than this, because now, sklearn `cross_val_predict` initialize the shape of `y_pred` array based on the number of samples and the number of classes. However, with `SlidingEstimator`, `y_pred` is of shape `(n_samples, n_times, n_classes)`.

This leaves us with options 2 or 3.
then we need cross_val_multipredict ...

I think I join you on the '...' 

Shall I try to get this for the release, or is it better to give us some time to think?

The quick fix for now is to do e.g.

```python
class MyClf(LogisticRegression):
    def predict(self, X):
         return self.predict_proba(X)

cross_val_predict(MyClf(), X, y))
```
More generally, I think `cross_val_multipredict` would be useful. (Not volunteering :D)

Resolving patch:
diff --git a/mne/decoding/search_light.py b/mne/decoding/search_light.py
--- a/mne/decoding/search_light.py
+++ b/mne/decoding/search_light.py
@@ -279,6 +279,14 @@ def score(self, X, y):
         score = np.concatenate(score, axis=0)
         return score
 
+    @property
+    def classes_(self):
+        if not hasattr(self.estimators_[0], 'classes_'):
+            raise AttributeError('classes_ attribute available only if '
+                                 'base_estimator has it, and estimator %s does'
+                                 ' not' % (self.estimators_[0],))
+        return self.estimators_[0].classes_
+
 
 def _sl_fit(estimator, X, y, **fit_params):
     """Aux. function to fit SlidingEstimator in parallel.


Test patch:
diff --git a/mne/decoding/tests/test_search_light.py b/mne/decoding/tests/test_search_light.py
--- a/mne/decoding/tests/test_search_light.py
+++ b/mne/decoding/tests/test_search_light.py
@@ -2,7 +2,6 @@
 #
 # License: BSD (3-clause)
 
-
 import numpy as np
 from numpy.testing import assert_array_equal, assert_equal
 import pytest
@@ -236,3 +235,38 @@ def test_generalization_light():
         features_shape = pipe.estimators_[0].steps[0][1].features_shape_
         assert_array_equal(features_shape, [3, 4])
     assert_array_equal(y_preds[0], y_preds[1])
+
+
+@requires_version('sklearn', '0.17')
+def test_cross_val_predict():
+    """Test cross_val_predict with predict_proba."""
+    from sklearn.linear_model import LinearRegression
+    from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
+    from sklearn.base import BaseEstimator, clone
+    from sklearn.model_selection import cross_val_predict
+    rng = np.random.RandomState(42)
+    X = rng.randn(10, 1, 3)
+    y = rng.randint(0, 2, 10)
+
+    estimator = SlidingEstimator(LinearRegression())
+    cross_val_predict(estimator, X, y, cv=2)
+
+    class Classifier(BaseEstimator):
+        """Moch class that does not have classes_ attribute."""
+
+        def __init__(self):
+            self.base_estimator = LinearDiscriminantAnalysis()
+
+        def fit(self, X, y):
+            self.estimator_ = clone(self.base_estimator).fit(X, y)
+            return self
+
+        def predict_proba(self, X):
+            return self.estimator_.predict_proba(X)
+
+    with pytest.raises(AttributeError, match="classes_ attribute"):
+        estimator = SlidingEstimator(Classifier())
+        cross_val_predict(estimator, X, y, method='predict_proba', cv=2)
+
+    estimator = SlidingEstimator(LinearDiscriminantAnalysis())
+    cross_val_predict(estimator, X, y, method='predict_proba', cv=2)