Repository: mne-tools/mne-python

Issue:
cross_val_multiscore uses wrong cross-validation methods during running SlidingEstimator or GeneralizingEstimator in classification tasks
Based on the source code, **cross_val_multiscore** uses [`is_classifier(estimator)`](https://github.com/mne-tools/mne-python/blob/876192598da7c557dbf46a6dda2ced4b7088d4bb/mne/fixes.py#L1067) to assess the type of the input estimator and then create a cv object. 

[By default](https://github.com/mne-tools/mne-python/blob/master/mne/decoding/base.py#L436), if the input cv value is int (e.g. int(10)), and input estimator is a "classifier", it should create a **StratifiedKFold** cv object. 

### **However,**
when using `SlidingEstimator` or  `GeneralizingEstimator` this kind of "second order" estimators as the input estimator [(e.g. the MNE tutorial here)](http://www.nmr.mgh.harvard.edu/mne/dev/auto_tutorials/plot_sensors_decoding.html#temporal-decoding
), **cross_val_multiscore** can not correctly retrieve the estimator types.

This is because this kind of "second order" estimators do not have the `_estimator_type` attribute. 
This causes **cross_val_multiscore** misinterprets classifiers as regressors and creates an inappropriate cv object. 

By default, it creates a **KFold** (and without shuffling!!) object instead of **StratifiedKFold**.
This inappropriate cv may cause some serious problems in some situations (small sample size, trial has been sorted, ect.). 








Discussion:
hum indeed.

So SlidingEstimator and GeneralizingEstimator should be added the _
estimator_type attribute
as a property that's it right?

if so can you open a PR?

+1 for opening a PR

On Friday, 15 December 2017, Alexandre Gramfort <notifications@github.com>
wrote:

> hum indeed.
>
> So SlidingEstimator and GeneralizingEstimator should be added the _
> estimator_type attribute
> as a property that's it right?
>
> if so can you open a PR?
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/mne-tools/mne-python/issues/4834#issuecomment-351922620>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AEp7DFDNnjWABxzckje3nnMjVxuVsuRVks5tAg0hgaJpZM4RDBU6>
> .
>


Resolving patch:
diff --git a/mne/decoding/search_light.py b/mne/decoding/search_light.py
--- a/mne/decoding/search_light.py
+++ b/mne/decoding/search_light.py
@@ -39,6 +39,7 @@ class SlidingEstimator(BaseEstimator, TransformerMixin):
 
     def __init__(self, base_estimator, scoring=None, n_jobs=1):  # noqa: D102
         _check_estimator(base_estimator)
+        self._estimator_type = getattr(base_estimator, "_estimator_type", None)
         self.base_estimator = base_estimator
         self.n_jobs = n_jobs
         self.scoring = scoring


Test patch:
diff --git a/mne/decoding/tests/test_base.py b/mne/decoding/tests/test_base.py
--- a/mne/decoding/tests/test_base.py
+++ b/mne/decoding/tests/test_base.py
@@ -193,8 +193,8 @@ def test_linearmodel():
 @requires_version('sklearn', '0.18')
 def test_cross_val_multiscore():
     """Test cross_val_multiscore for computing scores on decoding over time."""
-    from sklearn.model_selection import KFold, cross_val_score
-    from sklearn.linear_model import LogisticRegression
+    from sklearn.model_selection import KFold, StratifiedKFold, cross_val_score
+    from sklearn.linear_model import LogisticRegression, LinearRegression
 
     # compare to cross-val-score
     X = np.random.rand(20, 3)
@@ -231,3 +231,20 @@ def test_cross_val_multiscore():
         clf.fit(X[train], y[train])
         scores_auc_manual.append(clf.score(X[test], y[test]))
     assert_array_equal(scores_auc, scores_auc_manual)
+
+    # indirectly test that cross_val_multiscore rightly detects the type of
+    # estimator and generates a StratifiedKFold for classiers and a KFold
+    # otherwise
+    X = np.random.randn(1000, 3)
+    y = np.r_[np.zeros(500), np.ones(500)]
+    clf = LogisticRegression(random_state=0)
+    reg = LinearRegression()
+    for cross_val in (cross_val_score, cross_val_multiscore):
+        manual = cross_val(clf, X, y, cv=StratifiedKFold(2))
+        auto = cross_val(clf, X, y, cv=2)
+        assert_array_equal(manual, auto)
+        assert_raises(ValueError, cross_val, clf, X, y, cv=KFold(2))
+
+        manual = cross_val(reg, X, y, cv=KFold(2))
+        auto = cross_val(reg, X, y, cv=2)
+        assert_array_equal(manual, auto)
diff --git a/mne/decoding/tests/test_search_light.py b/mne/decoding/tests/test_search_light.py
--- a/mne/decoding/tests/test_search_light.py
+++ b/mne/decoding/tests/test_search_light.py
@@ -29,13 +29,16 @@ def test_search_light():
     from sklearn.pipeline import make_pipeline
     from sklearn.metrics import roc_auc_score, make_scorer
     from sklearn.ensemble import BaggingClassifier
+    from sklearn.base import is_classifier
 
     X, y = make_data()
     n_epochs, _, n_time = X.shape
     # init
     assert_raises(ValueError, SlidingEstimator, 'foo')
     sl = SlidingEstimator(Ridge())
+    assert_true(not is_classifier(sl))
     sl = SlidingEstimator(LogisticRegression())
+    assert_true(is_classifier(sl))
     # fit
     assert_equal(sl.__repr__()[:18], '<SlidingEstimator(')
     sl.fit(X, y)