Repository: mne-tools/mne-bids-pipeline

Issue:
Making MNE-BIDS-Pipeline a proper Python package
It was a conscious design decision **not** to make the pipeline a Python package for the following reasons:

- The code shouldn't be "hidden" from the users, to facilitate looking at and potentially editing the scripts
- It should be possible to have multiple versions of the pipeline installed within the same Python environment, to make working on multiple projects in parallel easier

I believe it's safe to assume that:
- At least since we added support for parallelization and now caching, the complexity of our scripts has exploded and they're not trivial to understand to many users anymore. This is regrettable but apparently unavoidable if we want to have the features that are required to improve pipeline adoption.
- Even before complexity increased to the current level, contributions from folks outside of the "@agramfort bubble" were only marginal.
- It's trivially simple and IMHO good practice to create separate virtual Python environments for every project and analysis you're working on.
- Installation and updating of the pipeline is currently all but user-friendly and I believe a **major** blocker for pipeline adoption.
- The manual steps required to create a custom config file are painful and not well-documented or not documented at all, causing users to do "wrong" things like copying config.py and forgetting to remove all the clutter below that "custom config ends here" horizon, which leads to an infinite recursion when starting the pipeline. 

I had started working on the MNE-BIDS-Pipeline-Runner a while back, attempting to ease this pain; but I never had enough time to bring this project to a usable state:

https://github.com/hoechenberger/mne-bids-pipeline-runner

Ultimately, it was just a workaround for the fact that the pipeline was never available as a package. But I like the idea of providing an interactive prompt that just asks you what to do. 

But with all of the above being said -- I believe it's finally time to:

- Make the pipeline a proper Python package
- Add an entry point to run the pipeline from the command line
- Add an entry point to create a custom / template config. 
- Expose the functionality from `run.py::process()` nicely such that users can invoke the pipeline from their scripts. This bit invaluable to me when it comes to debugging. In fact, every time I wish to debug the pipeline, I create an `mwe.py` inside the pipeline directory, which imports `process` from `run.py`. Only that way can I use the VS Code debugger that I'm used to work with. 

A package could also be installed in "editable" mode, so interested parties could still dump the repository folder wherever they want on their hard drive and edit / look at the code at will. 

I'd like to hear your opinions on the thoughts and proposals above 👍


Discussion:
This seems to approach the functionality of nipype and ... to some extent neuropycon. I hope we're not re-inventing the wheel here? This idea has been tried before and probably never flew -> the reason I think is that neuroimaging pipelines for bleeding edge MEG research are hard to put in a box. I struggle even with the flexibility of MNE-Python

IMO ... if you can create BIDS-apps for specific well-defined steps of the pipeline, that is a bigger win. A BIDS-app for maxfilter? If I have 300 subjects in BIDS format on AWS, can I point to that directory and get maxfiltered output files in a few minutes? 
I'm failing to see how this contradicts making it a package? You can put it into a Docker container and that's basically it. In fact, you can do that today already ... making it a package with proper entry points actually will make it easier to have things installed & run through Docker.

But maybe I just don't understand your point 🤔 Could you elaborate how making the pipeline a package would "hinder" the idea of a BIDS app?

Re 300 subjects with MaxFilter -- we've processed some 650 using the MNE-BIDS-Pipeline several times already (Cam-CAN) -- flawless approach utilizing all 80-something CPU cores of our system. 
I guess the goal of making a package relates to what the goals of MNE-BIDS-pipeline are ... when we first released the code, it was to give the community clear guidelines and easy example code to run MEG analysis. If you make it a package, the goal shifts towards giving the community "lego blocks" from which to construct more complex pipelines. You will need to cache intermediate results and handle errors gracefully. I am guessing you would need to create an API ... and that begins to approach the functionality of the above-mentioned packages. 

Could you point me to the Cam-CAN BIDS app? Could I use it today to run 10 BIDS-formatted subjects on my workstation? I do not have sudo privileges ...
> The manual steps required to create a custom config file are painful and not well-documented or not documented at all, causing users to do "wrong" things like copying config.py and forgetting to remove all the clutter below that "custom config ends here" horizon, which leads to an infinite recursion when starting the pipeline.

Create a thin GUI that allows users to set `config.py` so they never touch code?
> I guess the goal of making a package relates to what the goals of MNE-BIDS-pipeline are ... when we first released the code, it was to give the community clear guidelines and easy example code to run MEG analysis. If you make it a package, the goal shifts towards giving the community "lego blocks" from which to construct more complex pipelines.

I don't get that point. All I'm saying is take what we have here and make it into a package so it's easier to install and use.

The code is already so complex that "ordinary" users will struggle to understand what's going on.

> You will need to cache intermediate results and handle errors gracefully. 

Caching has just been added in #563 

> Could you point me to the Cam-CAN BIDS app? Could I use it today to run 10 BIDS-formatted subjects on my workstation? 

There is no BIDS app, it's a config file to be used with the pipeline that @agramfort used a few times to process Cam-CAN. He hosts a repository of such configs, but I've forgotten the URL.

> I do not have sudo privileges ...

I don't see why you should need any?


> Create a thin GUI that allows users to set `config.py` so they never touch code?

Creation of a wizard for config generation was one of the points on my todo list of [MNE-BID-Pipeline-Runner](https://github.com/hoechenberger/mne-bids-pipeline-runner), but I never had enough time to implement this bit.

Still doesn't solve the issue of installation, which would be greatly simplified if we had a package.
To me MNE-BIDS-Pipeline is already very close to a package, and there are more advantages than drawbacks to converting it to one. So +1 for the top-comment plan from me
Let's discuss this Friday if we find some time during the office hours? (I cannot promise anything though)
honestly we should discuss this again.

Let's first start by doing a better documentation about config extensions
Just to add a bit here, I think making this a package could greatly facilitate adoption. We can package it with the MNE installers and tell people they can `mne_bids_pipeline my_config.py` and if everything is setup correctly end up with fully preprocessed data with reporting, caching, etc. I think this should be a goal for the next ~3 months / for the 1.3 release

Resolving patch:
diff --git a/__init__.py b/__init__.py
new file mode 100644
--- /dev/null
+++ b/__init__.py
@@ -0,0 +1,7 @@
+from importlib.metadata import version, PackageNotFoundError
+
+try:
+    __version__ = version("mne_bids_pipeline")
+except PackageNotFoundError:
+    # package is not installed
+    pass
diff --git a/run.py b/run.py
--- a/run.py
+++ b/run.py
@@ -316,5 +316,9 @@ def process(
         logger.info('Done running 👆', extra=extra)
 
 
-if __name__ == '__main__':
+def main():
     fire.Fire(process)
+
+
+if __name__ == '__main__':
+    main()


Test patch:
diff --git a/tests/environment.yml b/tests/environment.yml
deleted file mode 100644
--- a/tests/environment.yml
+++ /dev/null
@@ -1,38 +0,0 @@
-
-# NOTE: This file is currently not used and has been superseded by
-#       requirements.txt
-#
-# name: base
-# channels:
-#   - conda-forge
-# dependencies:
-#   - python =3.8
-#   - typing_extensions
-#   - openneuro-py
-#   - datalad
-#   - numpy
-#   - scipy
-#   - scikit-learn
-#   - pandas
-#   - json_tricks
-#   - matplotlib
-#   - seaborn
-#   - nibabel
-#   - coloredlogs
-#   - python-picard
-#   - fire
-#   - pyqt
-#   - pyvista
-#   - pyvistaqt
-#   - pip
-#   - mkdocs
-#   - mkdocs-material
-#   - mkdocs-material-extensions
-#   - mkdocstrings
-#   - mkdocs-macros-plugin
-#   - coloredlogs
-#   - autoreject
-#   - pip:
-#     - http://github.com/hoechenberger/openneuro-py/zipball/main
-#     - https://api.github.com/repos/mne-tools/mne-python/zipball/main
-#     - https://api.github.com/repos/mne-tools/mne-bids/zipball/main
diff --git a/tests/requirements.txt b/tests/requirements.txt
deleted file mode 100644
--- a/tests/requirements.txt
+++ /dev/null
@@ -1,15 +0,0 @@
-psutil
-datalad
-mkdocs
-mkdocs-material
-mkdocs-material-extensions
-mkdocs-macros-plugin
-mkdocstrings-python
-jinja2 < 3.1.0  # https://twitter.com/readthedocs/status/1507388916013314048
-black  # function signature formatting
-livereload
-openneuro-py
-httpx >= 0.20
-tqdm
-Pygments <= 2.11.2
-pyyaml
diff --git a/tests/run_tests.py b/tests/run_tests.py
--- a/tests/run_tests.py
+++ b/tests/run_tests.py
@@ -5,17 +5,16 @@
 import os
 from pathlib import Path
 import argparse
-import runpy
 from typing import Collection, Dict, Optional
 if sys.version_info >= (3, 8):
     from typing import TypedDict
 else:
     from typing_extensions import TypedDict
 
+from mne.utils import run_subprocess
 
-# Add the pipelines dir to the PATH
-study_template_dir = Path(__file__).absolute().parents[1]
-sys.path.append(str(study_template_dir))
+
+BIDS_PIPELINE_DIR = Path(__file__).absolute().parents[1]
 
 
 def fetch(dataset=None):
@@ -152,7 +151,7 @@ def run_tests(test_suite, *, download, debug, cache):
             os.environ.update(test_options['env'])
 
         config = test_options.get('config', f'config_{dataset}.py')
-        config_path = study_template_dir / 'tests' / 'configs' / config
+        config_path = BIDS_PIPELINE_DIR / 'tests' / 'configs' / config
 
         # Fetch the data.
         dataset_name = test_options.get('dataset', dataset.split('_')[0])
@@ -175,7 +174,6 @@ def run_tests(test_suite, *, download, debug, cache):
                 dst=Path('~/mne_data/ds001971/participants.tsv').expanduser()
             )
 
-
         # Test the `--n_jobs` parameter
         if dataset == 'ds000117':
             n_jobs = '1'
@@ -186,30 +184,18 @@ def run_tests(test_suite, *, download, debug, cache):
         steps = test_options.get(
             'steps', ('preprocessing', 'sensor', 'report'))
         task = test_options.get('task', None)
-
-        run_script = study_template_dir / 'run.py'
-        # We need to adjust sys.argv so we can pass "command line arguments"
-        # to run.py when executed via runpy.
-        argv_orig = sys.argv.copy()
-        run_path = str(run_script)
-        sys.argv = [
-            run_path,
-            f'--steps={",".join(steps)}',
+        command = [
+            'mne_bids_pipeline',
             f'--config={config_path}',
+            f'--steps={",".join(steps)}',
             f'--task={task}' if task else '',
             f'--n_jobs={n_jobs}' if n_jobs else '',
             '--debug=1' if debug else '',
             '--cache=0' if not cache else '',
             f'--interactive=0'
         ]
-        # Eliminate "empty" items
-        sys.argv = [arg for arg in sys.argv if arg != '']
-        # We have to use run_path because run_module doesn't allow
-        # relative imports.
-        try:
-            runpy.run_path(run_path, run_name='__main__')
-        finally:
-            sys.argv = argv_orig
+        command = [x for x in command if x != ''] # Eliminate "empty" items
+        run_subprocess(command=command)
 
 
 if __name__ == '__main__':
@@ -267,3 +253,4 @@ def run_tests(test_suite, *, download, debug, cache):
           f'{", ".join(test_suite.keys())}')
 
     run_tests(test_suite, download=download, debug=debug, cache=cache)
+