Repository: mne-tools/mne-bids-pipeline

Issue:
StratifiedKFold with shuffling in the decoding step?
Currently we create the cross-validation object in our decoding step (08) of the pipeline via:
https://github.com/mne-tools/mne-study-template/blob/b61a5ca66aaef1f631d7ce2def3b1cde5d611729/08-sliding_estimator.py#L80-L81

By default, `StratifiedKFold` does not shuffle, meaning that the passed `random_state` doesn't have any effect (it produces a warning, though).

So – should we enable shuffling? Intuitively I would say yes, but want to hear your opinion, @agramfort


Discussion:
we could. But I would even favor using StratifiedShuffleSplit

https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedShuffleSplit.html

but we should do this after showing empirical impact on a few datasets

>

do we want shuffling? nearby epochs could make the decoding easier for the classifier? e.g., if there was a temporary dc offset
true that can be a bad idea if you have short ISI

Maybe we shouldn't create our own `cv` object in the first place and simply go with the `mne-python` defaults instead?

https://mne.tools/dev/generated/mne.decoding.cross_val_multiscore.html#mne.decoding.cross_val_multiscore

> cv int, cross-validation generator | iterable
> 
> Determines the cross-validation splitting strategy. Possible inputs for cv are:
> None, to use the default 3-fold cross validation,
> integer, to specify the number of folds in a (Stratified)KFold,
> An object to be used as a cross-validation generator.
> An iterable yielding train, test splits.
> For integer/None inputs, if the estimator is a classifier and y is either binary or multiclass, sklearn.model_selection.StratifiedKFold is used. In all other cases, sklearn.model_selection.KFold is used.
> 

Resolving patch:
diff --git a/08-sliding_estimator.py b/08-sliding_estimator.py
--- a/08-sliding_estimator.py
+++ b/08-sliding_estimator.py
@@ -77,9 +77,7 @@ def run_time_decoding(subject, condition1, condition2, session=None):
                       LogisticRegression(solver='liblinear',
                                          random_state=config.random_state)),
         scoring=config.decoding_metric, n_jobs=config.N_JOBS)
-    cv = StratifiedKFold(random_state=config.random_state,
-                         n_splits=config.decoding_n_splits)
-    scores = cross_val_multiscore(se, X=X, y=y, cv=cv)
+    scores = cross_val_multiscore(se, X=X, y=y, cv=config.decoding_n_splits)
 
     # let's save the scores now
     a_vs_b = '%s_vs_%s' % (condition1, condition2)
diff --git a/11-make_cov.py b/11-make_cov.py
--- a/11-make_cov.py
+++ b/11-make_cov.py
@@ -50,13 +50,8 @@ def compute_cov_from_epochs(subject, session, tmin, tmax):
                                 session=session))
 
     epochs = mne.read_epochs(epo_fname, preload=True)
-
-    # Do not shuffle the data before splitting into train and test samples.
-    # Perform a block cross-validation instead to maintain autocorrelated
-    # noise.
-    cv = KFold(3, shuffle=False)
     cov = mne.compute_covariance(epochs, tmin=tmin, tmax=tmax, method='shrunk',
-                                 cv=cv, rank='info')
+                                 rank='info')
     cov.save(cov_fname)
 
 
@@ -88,13 +83,7 @@ def compute_cov_from_empty_room(subject, session):
                                 session=session))
 
     raw_er = mne.io.read_raw_fif(raw_er_fname, preload=True, **extra_params)
-
-    # Do not shuffle the data before splitting into train and test samples.
-    # Perform a block cross-validation instead to maintain autocorrelated
-    # noise.
-    cv = KFold(3, shuffle=False)
-    cov = mne.compute_raw_covariance(raw_er, method='shrunk', cv=cv,
-                                     rank='info')
+    cov = mne.compute_raw_covariance(raw_er, method='shrunk', rank='info')
     cov.save(cov_fname)
 
 


Test patch: