Repository: mne-tools/mne-bids-pipeline

Issue:
MNE-BIDS-Pipeline maxwell filter reference run error
https://mne.discourse.group/t/mne-bids-pipeline-maxwell-filter-reference-run-error/7030

Hi, 

I am getting the below error whenever I set a run number different than '01' in the maxwell filter reference run parameter (```mf_reference_run```) in mne-bids-pipeline. 
```
[16:19:22] │ ⏳️ preprocessing/_01_data_quality sub-02 run-02 Finding flat channels and noisy channels using Maxwell filtering.
Opening raw data file /imaging/davis/Projects/SpeechMisperceptionMEEG/MNE-BIDS_data/sub-02/meg/sub-02_task-main_run-02_meg.fif...
joblib.externals.loky.process_executor._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/ma09/.conda/envs/my_mne1.4/lib/python3.11/site-packages/mne_bids_pipeline/_run.py", line 54, in wrapper
    out = memory.cache(func)(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ma09/.conda/envs/my_mne1.4/lib/python3.11/site-packages/mne_bids_pipeline/_run.py", line 147, in wrapper
    in_files = kwargs["in_files"] = self.get_input_fnames(**these_kwargs)
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ma09/.conda/envs/my_mne1.4/lib/python3.11/site-packages/mne_bids_pipeline/steps/preprocessing/_01_data_quality.py", line 46, in get_input_fnames_data_quality
    in_files = _get_raw_paths(
               ^^^^^^^^^^^^^^^
  File "/home/ma09/.conda/envs/my_mne1.4/lib/python3.11/site-packages/mne_bids_pipeline/_import_data.py", line 564, in _get_raw_paths
    raw_fname = _read_json(_empty_room_match_path(bids_path_in, cfg))[
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ma09/.conda/envs/my_mne1.4/lib/python3.11/site-packages/mne_bids_pipeline/_io.py", line 17, in _read_json
    with open(fname, "r", encoding="utf-8") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/imaging/davis/Projects/SpeechMisperceptionMEEG/data/derivatives/mne-bids-pipeline/sub-01/meg/sub-01_task-main_run-01_emptyroommatch.json'
```
So the pipeline is looking for a file named 'sub-01_task-main_run-01_emptyroommatch.json', whereas the actual file is called sub-01_task-main_run-03_emptyroommatch.json as I set ```mf_reference_run = '03'``` as it is recommended that a middle run should be used in this situation.


Discussion:
Hello! 👋 Thanks for opening your first issue here! ❤️ We will try to get back to you soon. 🚴🏽‍♂️

@allermat do you have `use_maxwell_filter = True` in your config? It seems like this should make it use the ref run:

https://github.com/mne-tools/mne-bids-pipeline/blob/3e8d5fdc528d9dec547b941ed55fa5847ca1b835/mne_bids_pipeline/steps/init/_02_find_empty_room.py#L133-L136

But maybe later we don't have the correct logic. And even if you set `use_maxwell_filter = False` it shouldn't error like this I think. So definitely some bug to fix. @allermat also feel free to try fixing the code and opening a PR if you're up for it!

Resolving patch:
diff --git a/mne_bids_pipeline/_import_data.py b/mne_bids_pipeline/_import_data.py
--- a/mne_bids_pipeline/_import_data.py
+++ b/mne_bids_pipeline/_import_data.py
@@ -561,9 +561,17 @@ def _get_raw_paths(
                 if task == "rest":
                     raw_fname = bids_path_in.copy().update(run=None, task=task)
                 else:
-                    raw_fname = _read_json(_empty_room_match_path(bids_path_in, cfg))[
-                        "fname"
-                    ]
+                    # This must match the logic of _02_find_empty_room.py
+                    update_kwargs = dict()
+                    if cfg.use_maxwell_filter:
+                        update_kwargs["run"] = cfg.mf_reference_run
+                    raw_fname = _read_json(
+                        _empty_room_match_path(
+                            bids_path_in.copy().update(**update_kwargs),
+                            cfg,
+                        )
+                    )
+                    raw_fname = raw_fname["fname"]
                     if raw_fname is not None:
                         raw_fname = get_bids_path_from_fname(raw_fname)
             if raw_fname is None:
diff --git a/mne_bids_pipeline/steps/preprocessing/_01_data_quality.py b/mne_bids_pipeline/steps/preprocessing/_01_data_quality.py
--- a/mne_bids_pipeline/steps/preprocessing/_01_data_quality.py
+++ b/mne_bids_pipeline/steps/preprocessing/_01_data_quality.py
@@ -292,7 +292,6 @@ def get_config(
             subject=subject,
             session=session,
         )
-        extra_kwargs["mf_reference_run"] = get_mf_reference_run(config=config)
         extra_kwargs["mf_head_origin"] = config.mf_head_origin
     cfg = SimpleNamespace(
         process_empty_room=config.process_empty_room,
@@ -322,6 +321,8 @@ def get_config(
         min_break_duration=config.min_break_duration,
         t_break_annot_start_after_previous_event=config.t_break_annot_start_after_previous_event,  # noqa:E501
         t_break_annot_stop_before_next_event=config.t_break_annot_stop_before_next_event,  # noqa:E501
+        use_maxwell_filter=config.use_maxwell_filter,
+        mf_reference_run=get_mf_reference_run(config=config),
         data_type=config.data_type,
         ch_types=config.ch_types,
         eog_channels=config.eog_channels,


Test patch:
diff --git a/mne_bids_pipeline/tests/configs/config_ds000117.py b/mne_bids_pipeline/tests/configs/config_ds000117.py
--- a/mne_bids_pipeline/tests/configs/config_ds000117.py
+++ b/mne_bids_pipeline/tests/configs/config_ds000117.py
@@ -17,11 +17,11 @@
 raw_resample_sfreq = 125.0
 crop_runs = (0, 300)  # Reduce memory usage on CI system
 
-find_flat_channels_meg = False
-find_noisy_channels_meg = False
+find_flat_channels_meg = True
+find_noisy_channels_meg = True
 use_maxwell_filter = True
 
-mf_reference_run = "01"
+mf_reference_run = "02"
 mf_cal_fname = bids_root + "/derivatives/meg_derivatives/sss_cal.dat"
 mf_ctc_fname = bids_root + "/derivatives/meg_derivatives/ct_sparse.fif"
 
diff --git a/mne_bids_pipeline/tests/conftest.py b/mne_bids_pipeline/tests/conftest.py
--- a/mne_bids_pipeline/tests/conftest.py
+++ b/mne_bids_pipeline/tests/conftest.py
@@ -31,6 +31,8 @@ def pytest_configure(config):
     ignore:_SixMetaPathImporter\.find_spec.*:ImportWarning
     ignore:pkg_resources is deprecated.*:DeprecationWarning
     ignore:`product` is deprecated as of NumPy.*:DeprecationWarning
+    # seaborn calling tight layout
+    ignore:The figure layout has changed to tight:UserWarning
     """
     for warning_line in warning_lines.split("\n"):
         warning_line = warning_line.strip()