Repository: mne-tools/mne-bids-pipeline

Issue:
How to deploy for multiple analyses
In the case where one would like to run the pipeline

- on the same dataset multiple times with different settings, or
- on different datasets

it is unclear to me what's the best practice regarding `mne-study-template` deployment.

While the readme [suggests](https://github.com/mne-tools/mne-study-template/blob/082a959444b464603eb4f5ed5404bf19b872a1d5/README.md#running-on-your-own-data) (point 3) that one may define an `MNE_BIDS_STUDY_CONFIG` environment variable to use one global installation of the pipeline with different configurations, [the code](https://github.com/mne-tools/mne-study-template/blob/082a959444b464603eb4f5ed5404bf19b872a1d5/config.py#L707-L709) makes it clear that this is really just intended for running the CI tests. So – what gives? Currently I have multiple copies of `mne-study-template`, each with their own `config.py`, to analyze different projects.

This seems cumbersome, and it's also related to the question @agramfort [recently raised](https://github.com/mne-tools/mne-study-template/pull/74#issuecomment-617253746): Should the pipeline be a loose collection of scripts, or maybe an installable package?

One issue that often arises with "global" packages / installations is that one is usually pinned to just _one_ package version. However, with the advance of virtual environments this should be less of an issue these days. But even so, one could make use of the [version identifier](https://github.com/mne-tools/mne-study-template/blob/082a959444b464603eb4f5ed5404bf19b872a1d5/config.py#L16) in `config.py` to automatically retrieve the correct pipeline version from GitHub. PsychoPy has a [similar functionality](https://github.com/psychopy/psychopy/blob/2d534f7fe2282e2a041972cdebfba3379e6fa8c5/psychopy/tools/versionchooser.py#L41) where it allows users to specify the exact package version the experiment should be run with.

What are your general thoughts on this matter?

tagging @jasmainak @agramfort


Discussion:
as discussed IRL you should add support for absolute path for config file

>

agree with @agramfort on this particular issue

regarding the possibility of giving users the ability to modify scripts, do you want to have that discussion here? 

Resolving patch:
diff --git a/config.py b/config.py
--- a/config.py
+++ b/config.py
@@ -722,9 +722,23 @@ def default_reject_comps_factory():
 # For example `export MNE_BIDS_STUDY_CONFIG=config_eeg_matchingpennies`
 
 if "MNE_BIDS_STUDY_CONFIG" in os.environ:
-    cfg_name = os.environ['MNE_BIDS_STUDY_CONFIG']
-    cfg_path = 'tests.configs.{}'.format(cfg_name)
-    custom_cfg = importlib.import_module(cfg_path)
+    cfg_path = os.environ['MNE_BIDS_STUDY_CONFIG']
+
+    if os.path.exists(cfg_path):
+        print('Using custom configuration specified in MNE_BIDS_STUDY_CONFIG.')
+    else:
+        msg = ('The custom configuration file specified in the '
+               'MNE_BIDS_STUDY_CONFIG environment variable could not be '
+               'found: {cfg_path}'.format(cfg_path=cfg_path))
+        raise ValueError(msg)
+
+    # Import configuration from an arbitrary path without having to fiddle
+    # with `sys.path`.
+    spec = importlib.util.spec_from_file_location(name='custom_config',
+                                                  location=cfg_path)
+    custom_cfg = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(custom_cfg)
+    del spec
 
     new = None
     for val in dir(custom_cfg):


Test patch:
diff --git a/tests/run_tests.py b/tests/run_tests.py
--- a/tests/run_tests.py
+++ b/tests/run_tests.py
@@ -6,7 +6,8 @@
 import importlib
 
 # Add the pipelines dir to the PATH
-sys.path.append(op.join(op.dirname(__file__), '..'))
+pipeline_dir = os.path.abspath(op.join(op.dirname(__file__), '..'))
+sys.path.append(pipeline_dir)
 
 
 def fetch(dataset=None):
@@ -92,7 +93,12 @@ def run_tests(test_suite):
         # export the environment variables
         os.environ['DATASET'] = dataset
         os.environ['BIDS_ROOT'] = op.join(DATA_DIR, dataset)
-        os.environ['MNE_BIDS_STUDY_CONFIG'] = test_tuple[0]
+
+        config_name = test_tuple[0]
+        config_path = os.path.join(pipeline_dir, 'tests', 'configs',
+                                   config_name + '.py')
+        os.environ['MNE_BIDS_STUDY_CONFIG'] = config_path
+        del config_name, config_path
 
         # Fetch the data
         fetch(dataset)