internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: mne-tools/mne-bids-pipeline

Issue:
config.crop and multiple runs
Current docs seem to indicate the `crop` setting applies to the concatenated data an entire session:

```
crop: Optional[Tuple[float, float]] = None
"""
Crop the raw data to the specified time interval ``[tmin, tmax]`` in seconds.
If ``None``, do not crop the data.
"""
```

However, we're actually cropping each run individually.

Either we
- keep the current behavior, but rename the setting to `crop_runs` or something similar,
- only apply cropping on concatenated data (i.e., after concatenating all runs) -> can be problematic in some cases if runs are long and we need to crop due to memory constraints!
- keep the current behavior but introduce an additional setting for cropping the concatenated data

WDYT?


Discussion:
+1 to crop all runs as it is

and yes let's rename to crop_runs to be more explicit

>


Resolving patch:
diff --git a/config.py b/config.py
--- a/config.py
+++ b/config.py
@@ -73,12 +73,6 @@
 window.
 """
 
-crop: Optional[Tuple[float, float]] = None
-"""
-Crop the raw data to the specified time interval ``[tmin, tmax]`` in seconds.
-If ``None``, do not crop the data.
-"""
-
 sessions: Union[List, Literal['all']] = 'all'
 """
 The sessions to process.
@@ -94,6 +88,13 @@
 The runs to process.
 """
 
+crop_runs: Optional[Tuple[float, float]] = None
+"""
+Crop the raw data of each run to the specified time interval ``[tmin, tmax]``,
+in seconds. The runs will be cropped before Maxwell or frequency filtering is
+applied. If ``None``, do not crop the data.
+"""
+
 acq: Optional[str] = None
 """
 The BIDS `acquisition` entity.
diff --git a/scripts/preprocessing/01-import_and_maxfilter.py b/scripts/preprocessing/01-import_and_maxfilter.py
--- a/scripts/preprocessing/01-import_and_maxfilter.py
+++ b/scripts/preprocessing/01-import_and_maxfilter.py
@@ -227,8 +227,8 @@ def load_data(bids_path):
 
     if subject != 'emptyroom':
         # Crop the data.
-        if config.crop is not None:
-            raw.crop(*config.crop)
+        if config.crop_runs is not None:
+            raw.crop(*config.crop_runs)
 
         # Rename events.
         if config.rename_events:


Test patch:
diff --git a/tests/configs/config_ds000117.py b/tests/configs/config_ds000117.py
--- a/tests/configs/config_ds000117.py
+++ b/tests/configs/config_ds000117.py
@@ -13,7 +13,7 @@
 subjects = ['01']
 
 resample_sfreq = 125.
-crop = (0, 350)
+crop_runs = (0, 350)  # Reduce memory usage on CI system
 
 find_flat_channels_meg = False
 find_noisy_channels_meg = False
Repository: mne-tools/mne-bids-pipeline

Issue:
Documentation build is currently broken



Discussion:

Resolving patch:
diff --git a/config.py b/config.py
--- a/config.py
+++ b/config.py
@@ -432,7 +432,7 @@
 
 t_break_annot_start_after_previous_event: float = 5.
 """
-Once a break of at least [`min_break_duration`][configmin_break_duration]
+Once a break of at least [`min_break_duration`][config.min_break_duration]
 seconds has been discovered, we generate a `BAD_break` annotation that does not
 necessarily span the entire break period. Instead, you will typically want to
 start it some time after the last event before the break period, as to not
@@ -1947,7 +1947,7 @@ def get_task() -> Optional[str]:
     env = os.environ
     if env.get('MNE_BIDS_STUDY_TASK'):
         task = env['MNE_BIDS_STUDY_TASK']
-        if task not in _valid_tasks:
+        if task not in _valid_tasks and 'MKDOCS' not in os.environ:
             raise ValueError(f'Invalid task. It can be: '
                              f'{", ".join(_valid_tasks)} but got: {task}')
 


Test patch:
Repository: mne-tools/mne-bids-pipeline

Issue:
FreeSurfer step doesn't respect deriv_root
Instead, it will write outputs to `bids_root/derivatives/freesurfer` no matter what.

Reported by @dengemann 


Discussion:

Resolving patch:
diff --git a/config.py b/config.py
--- a/config.py
+++ b/config.py
@@ -59,13 +59,30 @@
 The root of the derivatives directory in which the pipeline will store
 the processing results. If ``None``, this will be
 ``derivatives/mne-bids-pipeline`` inside the BIDS root.
+
+Note: Note
+    If specified and you wish to run the source analysis steps, you must
+    set [`subjects_dir`][config.subjects_dir] as well.
 """
 
 subjects_dir: Optional[PathLike] = None
 """
-Path to the directory that contains the MRI data files and their
-derivativesfor all subjects. Specifically, the ``subjects_dir`` is the
-$SUBJECTS_DIR used by the Freesurfer software.
+Path to the directory that contains the FreeSurfer reconstructions of all
+subjects. Specifically, this defines the ``SUBJECTS_DIR`` that is used by
+FreeSurfer.
+
+- When running the ``freesurfer`` processing step to create the
+  reconstructions from anatomical scans in the BIDS dataset, the
+  output will be stored in this directory.
+- When running the source analysis steps, we will look for the surfaces in this
+  directory and also store the BEM surfaces there.
+
+If ``None``, this will default to
+[`bids_root`][config.bids_root_root]`/derivatives/freesurfer/subjects`.
+
+Note: Note
+    This setting is required if you specify [`deriv_root`][config.deriv_root]
+    and want to run the source analysis steps.
 """
 
 interactive: bool = False
@@ -2035,10 +2052,21 @@ def get_ica_reject() -> Dict[str, float]:
 
 
 def get_fs_subjects_dir():
+    if not subjects_dir and deriv_root is not None:
+        # We do this check here (and not in our regular checks section) to
+        # avoid an error message when a user doesn't intend to run the source
+        # analysis steps anyway.
+        raise ValueError(
+            'When specifying a "deriv_root", you must also supply a '
+            '"subjects_dir".'
+        )
+
     if not subjects_dir:
         return get_bids_root() / 'derivatives' / 'freesurfer' / 'subjects'
     else:
-        return subjects_dir
+        return (pathlib.Path(subjects_dir)
+                .expanduser()
+                .resolve())
 
 
 def gen_log_message(message, step=None, subject=None, session=None,


Test patch:
diff --git a/tests/configs/config_ds000246.py b/tests/configs/config_ds000246.py
--- a/tests/configs/config_ds000246.py
+++ b/tests/configs/config_ds000246.py
@@ -5,6 +5,7 @@
 study_name = 'ds000246'
 bids_root = '~/mne_data/ds000246'
 deriv_root = '~/mne_data/ds000246/derivatives/mne-bids-pipeline'
+subjects_dir = '~/mne_data/ds000246/derivatives/freesurfer/subjects'
 runs = ['01']
 l_freq = .3
 h_freq = 100.
Repository: mne-tools/mne-bids-pipeline

Issue:
Parallelize ERP CORE runs
We currently run all ERP CORE analyses sequentially, but I believe we should be able to parallelize them quite easily.


Discussion:

Resolving patch:
diff --git a/config.py b/config.py
--- a/config.py
+++ b/config.py
@@ -80,7 +80,7 @@
   directory and also store the BEM surfaces there.
 
 If ``None``, this will default to
-[`bids_root`][config.bids_root_root]`/derivatives/freesurfer/subjects`.
+[`bids_root`][config.bids_root]`/derivatives/freesurfer/subjects`.
 
 Note: Note
     This setting is required if you specify [`deriv_root`][config.deriv_root]


Test patch:
diff --git a/tests/run_tests.py b/tests/run_tests.py
--- a/tests/run_tests.py
+++ b/tests/run_tests.py
@@ -234,17 +234,16 @@ def run_tests(test_suite, download):
         # We need to adjust sys.argv so we can pass "command line arguments"
         # to run.py when executed via runpy.
         argv_orig = sys.argv.copy()
-        for step in steps:
-            sys.argv = [
-                sys.argv[0],
-                f'{step}',
-                f'--config={config_path}',
-                f'--task={task}' if task else '',
-                f'--n_jobs={n_jobs}' if n_jobs else ''
-            ]
-            # We have to use run_path because run_module doesn't allow
-            # relative imports.
-            runpy.run_path(run_script, run_name='__main__')
+        sys.argv = [
+            sys.argv[0],
+            f'--steps={",".join(steps)}',
+            f'--config={config_path}',
+            f'--task={task}' if task else '',
+            f'--n_jobs={n_jobs}' if n_jobs else ''
+        ]
+        # We have to use run_path because run_module doesn't allow
+        # relative imports.
+        runpy.run_path(run_script, run_name='__main__')
         sys.argv = argv_orig
Repository: mne-tools/mne-bids-pipeline

Issue:
Use Agg Matplotlib backend in non-interactive mode
It's faster and more robust.


Discussion:

Resolving patch:
diff --git a/config.py b/config.py
--- a/config.py
+++ b/config.py
@@ -24,6 +24,8 @@
 import pandas as pd
 from openpyxl import load_workbook
 import json_tricks
+import matplotlib
+
 import mne
 import mne_bids
 from mne_bids import BIDSPath, read_raw_bids
@@ -1755,6 +1757,10 @@ def gen_log_kwargs(
 if 'MNE_BIDS_STUDY_INTERACTIVE' in os.environ:
     interactive = bool(int(os.environ['MNE_BIDS_STUDY_INTERACTIVE']))
 
+if not interactive:
+    matplotlib.use('Agg')  # do not open any window  # noqa
+
+
 ###############################################################################
 # CHECKS
 # ------
diff --git a/scripts/report/_01_make_reports.py b/scripts/report/_01_make_reports.py
--- a/scripts/report/_01_make_reports.py
+++ b/scripts/report/_01_make_reports.py
@@ -14,7 +14,6 @@
 from typing import Tuple, Union, Optional
 
 from scipy.io import loadmat
-import matplotlib
 
 import mne
 from mne.utils import BunchConst
@@ -25,8 +24,6 @@
 import config
 from config import gen_log_kwargs, on_error, failsafe_run
 
-matplotlib.use('Agg')  # do not open any window  # noqa
-
 logger = logging.getLogger('mne-bids-pipeline')
 
 Condition_T = Union[str, Tuple[str]]


Test patch:
Repository: mne-tools/mne-bids-pipeline

Issue:
Covariance plots are missing from report



Discussion:

Resolving patch:
diff --git a/config.py b/config.py
--- a/config.py
+++ b/config.py
@@ -3232,7 +3232,7 @@ def import_er_data(
     raw_er.info['bads'] = [ch for ch in bads if ch.startswith('MEG')]
 
     # Only keep MEG channels.
-    raw_er.pick_types(meg=True)
+    raw_er.pick_types(meg=True, exclude=[])
 
     # Save the data.
     if save:
diff --git a/scripts/report/_01_make_reports.py b/scripts/report/_01_make_reports.py
--- a/scripts/report/_01_make_reports.py
+++ b/scripts/report/_01_make_reports.py
@@ -472,6 +472,7 @@ def run_report_sensor(
         suffix='itc+condition+tfr',
         extension='.h5'
     )
+    fname_noise_cov = bids_path.copy().update(suffix='cov')
 
     ###########################################################################
     #
@@ -501,6 +502,19 @@ def run_report_sensor(
         **gen_log_kwargs(message=msg, subject=subject, session=session)
     )
 
+    if fname_noise_cov.fpath.exists():
+        msg = f'Reading noise covariance: {fname_noise_cov.fpath.name}'
+        logger.info(
+            **gen_log_kwargs(message=msg, subject=subject, session=session)
+        )
+        noise_cov = fname_noise_cov
+    else:
+        msg = 'No noise covariance matrix found, not rendering whitened data'
+        logger.info(
+            **gen_log_kwargs(message=msg, subject=subject, session=session)
+        )
+        noise_cov = None
+
     for condition, evoked in zip(conditions, evokeds):
         if cfg.analyze_channels:
             evoked.pick(cfg.analyze_channels)
@@ -520,6 +534,7 @@ def run_report_sensor(
         report.add_evokeds(
             evokeds=evoked,
             titles=title,
+            noise_cov=noise_cov,
             tags=tags
         )
 
@@ -671,9 +686,11 @@ def run_report_source(
                                      subject=subject, session=session))
         return report
 
+    fname_noise_cov = bids_path.copy().update(suffix='cov')
+
     ###########################################################################
     #
-    # Visualize the coregistration & inverse solutions.
+    # Visualize coregistration, noise covariance matrix, & inverse solutions.
     #
 
     if cfg.conditions is None:
@@ -708,6 +725,15 @@ def run_report_source(
         alpha=1
     )
 
+    msg = 'Rendering noise covariance matrix and corresponding SVD.'
+    logger.info(**gen_log_kwargs(message=msg,
+                                 subject=subject, session=session))
+    report.add_covariance(
+        cov=fname_noise_cov,
+        info=fname_info,
+        title='Noise covariance'
+    )
+
     for condition in conditions:
         msg = f'Rendering inverse solution for {condition}'
         logger.info(**gen_log_kwargs(message=msg,


Test patch:
Repository: mne-tools/mne-bids-pipeline

Issue:
Fix log message "Loading reference run: None"
```
[21:10:24] preprocessing/_01_maxfilter sub-01 Loading reference run: None.
[21:10:24] preprocessing/_01_maxfilter sub-01 Loading info for run: None.
```

This is ugly and should be fixed.


Discussion:

Resolving patch:
diff --git a/config.py b/config.py
--- a/config.py
+++ b/config.py
@@ -2267,6 +2267,12 @@ def get_runs_all_subjects() -> dict:
             get_bids_root() / f'sub-{subject}', entity_key='run',
             ignore_datatypes=tuple(_ignore_datatypes)
         )
+
+        # If we don't have any `run` entities, just set it to None, as we
+        # commonly do when creating a BIDSPath.
+        if not valid_runs_subj:
+            valid_runs_subj = [None]
+
         if exclude_runs and subject in exclude_runs:
             valid_runs_subj = [r for r in valid_runs_subj
                                if r not in exclude_runs[subject]]
@@ -2350,9 +2356,11 @@ def get_mf_reference_run() -> str:
         if inter_runs:
             return inter_runs[0]
         else:
-            ValueError("The intersection of runs by subjects is empty. "
-                       "Check the list of runs: "
-                       f"{get_runs_all_subjects()}")
+            raise ValueError(
+                f"The intersection of runs by subjects is empty. "
+                f"Check the list of runs: "
+                f"{get_runs_all_subjects()}"
+            )
     else:
         return mf_reference_run
 
diff --git a/scripts/preprocessing/_01_maxfilter.py b/scripts/preprocessing/_01_maxfilter.py
--- a/scripts/preprocessing/_01_maxfilter.py
+++ b/scripts/preprocessing/_01_maxfilter.py
@@ -58,9 +58,11 @@ def run_maxwell_filter(*, cfg, subject, session=None, run=None):
                              check=False)
 
     # Load dev_head_t and digitization points from MaxFilter reference run.
-    msg = f'Loading reference run: {cfg.mf_reference_run}.'
-    logger.info(**gen_log_kwargs(message=msg, subject=subject,
-                                 session=session, run=run))
+    if cfg.mf_reference_run is not None:
+        # Only log if we have more than just a single run
+        msg = f'Loading reference run: {cfg.mf_reference_run}.'
+        logger.info(**gen_log_kwargs(message=msg, subject=subject,
+                                     session=session, run=run))
 
     reference_run_params = get_reference_run_params(
         subject=subject, session=session, run=cfg.mf_reference_run


Test patch:
Repository: mne-tools/mne-bids-pipeline

Issue:
Lowpass filter at 40Hz is hardcoded in the config.py file
In the preprocessing, during the first script (`maxfilter`), raw data is imported with the function `import_experimental_data`. But in this function, there is another function called `_find_bad_channels` in which the raw data is filtered with a lowpass at 40Hz. If I choose a higher h_freq in the config it is not taken into account.

Here is the link to this specific line:
https://github.com/mne-tools/mne-bids-pipeline/blob/main/config.py#L3168




Discussion:
OUCH! Good catch!! This should operate on a **copy** of `raw` instead of on the original! Do you want to make a PR?

Resolving patch:
diff --git a/config.py b/config.py
--- a/config.py
+++ b/config.py
@@ -3167,18 +3167,21 @@ def _find_bad_channels(cfg, raw, subject, session, task, run) -> None:
                          root=cfg.deriv_root)
 
     # Filter the data manually before passing it to find_bad_channels_maxwell()
-    # This reduces memory usage!
-    raw.filter(l_freq=None, h_freq=40, n_jobs=1)
+    # This reduces memory usage, as we can control the number of jobs used
+    # during filtering.
+    raw_filt = raw.copy().filter(l_freq=None, h_freq=40, n_jobs=1)
+
     auto_noisy_chs, auto_flat_chs, auto_scores = \
         mne.preprocessing.find_bad_channels_maxwell(
-            raw=raw,
+            raw=raw_filt,
             calibration=cfg.mf_cal_fname,
             cross_talk=cfg.mf_ctc_fname,
             origin=mf_head_origin,
             coord_frame='head',
             return_scores=True,
-            h_freq=None  # We filtered manually before! So don't filter again.
+            h_freq=None  # we filtered manually above
         )
+    del raw_filt
 
     preexisting_bads = raw.info['bads'].copy()
     bads = preexisting_bads.copy()
@@ -3309,7 +3312,6 @@ def _create_bipolar_channels(cfg, raw, subject, session, run) -> None:
         logger.info(**gen_log_kwargs(
             message=msg, subject=subject, session=session, run=run)
         )
-        raw.load_data()
         for ch_name, (anode, cathode) in cfg.eeg_bipolar_channels.items():
             msg = f'    {anode} – {cathode} -> {ch_name}'
             logger.info(**gen_log_kwargs(
@@ -3360,7 +3362,10 @@ def _set_eeg_montage(cfg, raw, subject, session, run) -> None:
         raw.set_montage(montage, match_case=False, on_missing='warn')
 
 
-def _fix_stim_artifact_func(cfg: dict, raw: mne.io.BaseRaw) -> None:
+def _fix_stim_artifact_func(
+    cfg: SimpleNamespace,
+    raw: mne.io.BaseRaw
+) -> None:
     """Fix stimulation artifact in the data."""
     if not cfg.fix_stim_artifact:
         return


Test patch:
Repository: mne-tools/mne-bids-pipeline

Issue:
Fix Dask support
#547 disabled Dask in our CI tests, as recent changes in MNE-Python lead to a resource oversubscription. We need to take a closer look to figure out how to adjust the pipeline accordingly.

cc @agramfort @larsoner 


Discussion:

Resolving patch:
diff --git a/config.py b/config.py
--- a/config.py
+++ b/config.py
@@ -3475,8 +3475,7 @@ def _load_data(cfg, bids_path):
         picks = get_channels_to_analyze(raw.info)
         raw.pick(picks)
 
-    if bids_path.subject != 'emptyroom':
-        _crop_data(cfg, raw=raw, subject=subject)
+    _crop_data(cfg, raw=raw, subject=subject)
 
     raw.load_data()
     if hasattr(raw, 'fix_mag_coil_types'):


Test patch:
diff --git a/tests/configs/config_ERP_CORE.py b/tests/configs/config_ERP_CORE.py
--- a/tests/configs/config_ERP_CORE.py
+++ b/tests/configs/config_ERP_CORE.py
@@ -71,18 +71,14 @@
 on_error = 'abort'
 on_rename_missing_events = 'warn'
 
-# XXX temporarily disable Dask
-#
-# parallel_backend = 'dask'
-# dask_worker_memory_limit = '2G'
-# N_JOBS = 2
+parallel_backend = 'dask'
+dask_worker_memory_limit = '2G'
+N_JOBS = 2
 
 N_JOBS = 2
 
 if task == 'N400':
-    # XXX temporarily disable Dask
-    #
-    # dask_open_dashboard = True
+    dask_open_dashboard = True
 
     rename_events = {
         'response/201': 'response/correct',
diff --git a/tests/configs/config_ds000246.py b/tests/configs/config_ds000246.py
--- a/tests/configs/config_ds000246.py
+++ b/tests/configs/config_ds000246.py
@@ -1,5 +1,8 @@
 """
-Auditory MEG
+Brainstorm - Auditory Dataset.
+
+See https://openneuro.org/datasets/ds000246/versions/1.0.0 for more
+information.
 """
 
 study_name = 'ds000246'
@@ -7,6 +10,7 @@
 deriv_root = '~/mne_data/derivatives/mne-bids-pipeline/ds000246'
 
 runs = ['01']
+crop_runs = (0, 180)  # Reduce memory usage on CI system
 l_freq = 0.3
 h_freq = 100
 decim = 4
@@ -20,12 +24,8 @@
 decoding_time_generalization_decim = 4
 on_error = 'abort'
 
-# XXX temporarily disable Dask
-#
-# parallel_backend = 'dask'
-# dask_worker_memory_limit = '3G'
-# dask_temp_dir = "./.dask-worker-space"
-# dask_open_dashboard = True
-# N_JOBS = 2
-
+parallel_backend = 'dask'
+dask_worker_memory_limit = '2G'
+dask_temp_dir = "./.dask-worker-space"
+dask_open_dashboard = True
 N_JOBS = 2
diff --git a/tests/configs/config_ds001971.py b/tests/configs/config_ds001971.py
--- a/tests/configs/config_ds001971.py
+++ b/tests/configs/config_ds001971.py
@@ -1,14 +1,6 @@
-"""Configuration file for the ds001971 dataset.
-
-Set the `MNE_BIDS_STUDY_CONFIG` environment variable to
-"config_ds001971" to overwrite `config.py` with the values specified
-below.
-
-Download ds001971 from OpenNeuro: https://github.com/OpenNeuroDatasets/ds001971
-
-export MNE_BIDS_STUDY_CONFIG=config_ds001971
-export BIDS_ROOT=~/mne_data/ds001971
+"""Mobile brain body imaging (MoBI) gait adaptation experiment.
 
+See ds001971 on OpenNeuro: https://github.com/OpenNeuroDatasets/ds001971
 """
Repository: mne-tools/mne-bids-pipeline

Issue:
BUG: Resource oversubscription in ERP_CORE ICA on M1
I just ran with `n_jobs` set to 1 and this actually finished faster than with `2`, yet still utilized all CPU cores. Definitely resource oversubscription.

We're running the ERP CORE tests through Dask, if I'm not mistaken. So the problem might lie there…

_Originally posted by @hoechenberger in https://github.com/mne-tools/mne-bids-pipeline/pull/622#discussion_r990508420_
      


Discussion:

Resolving patch:
diff --git a/mne_bids_pipeline/run.py b/mne_bids_pipeline/run.py
--- a/mne_bids_pipeline/run.py
+++ b/mne_bids_pipeline/run.py
@@ -331,6 +331,10 @@ def process(
 
 
 def main():
+    # Fire does not seem to detect a "--help" in all locations, so let's do it
+    # manually.
+    if '--help' in sys.argv or '-h' in sys.argv:
+        sys.argv = sys.argv[:1] + ['--help']
     fire.Fire(process)
 
 
diff --git a/mne_bids_pipeline/scripts/preprocessing/_04a_run_ica.py b/mne_bids_pipeline/scripts/preprocessing/_04a_run_ica.py
--- a/mne_bids_pipeline/scripts/preprocessing/_04a_run_ica.py
+++ b/mne_bids_pipeline/scripts/preprocessing/_04a_run_ica.py
@@ -15,8 +15,10 @@
 from typing import List, Optional, Iterable, Tuple
 from types import SimpleNamespace
 
+from packaging.version import Version
 import pandas as pd
 import numpy as np
+import threadpoolctl
 
 import mne
 from mne.report import Report
@@ -267,6 +269,12 @@ def run_ica(*, cfg, subject, session, in_files):
         processing='ica+components', suffix='report', extension='.html')
     del bids_basename
 
+    # Prevent oversubscription of CPUs in dask on M1 on older dask
+    if cfg.parallel_backend == 'dask':
+        import dask
+        if Version(dask.__version__) < Version('2022.10.0'):
+            threadpoolctl.threadpool_limits(limits=1, user_api='openmp')
+
     # Generate a list of raw data paths (i.e., paths of individual runs)
     # we want to create epochs from.
 
@@ -554,6 +562,7 @@ def get_config(
         eog_channels=config.eog_channels,
         rest_epochs_duration=config.rest_epochs_duration,
         rest_epochs_overlap=config.rest_epochs_overlap,
+        parallel_backend=config.parallel_backend,
     )
     return cfg
 


Test patch: