Repository: nilearn/nilearn

Issue:
nilearn.interfaces.fmriprep.load_confounds - Allow users to specify confounds filename directly
<!--Provide a brief description what you would like changes and why.-->
Allow users of nilearn.interfaces.fmriprep.load_confounds to specify the confounds filename (and associated metadata, if necessary) directly rather than requiring that the confounds filename be inferred from the fMRI filename.

<!--Please fill in the following information, to the best of your ability.-->
### Benefits to the change
1. Allow users to be in control of the which confounds file is loaded.
2. The imaging file isn't necessary for loading the confounds.

### Pseudocode for the new behavior, if applicable

```python
load_confounds(confounds_filename=my_confounds_file, confounds_metadata=my_confounds_metadata_file, **kwargs)
```


I'm willing to take a stab at this if this request is approved.


Discussion:
@ellisdg this is actually an intentional design, as ICA-AROMA based strategy will require the user to use a different preprocessed file. If the dataset is preprocessed by fMRIPrep correctly without any external intervention, a image files should be stored under the same directory with the associated confound regressors.
If you look into the earlier version of [`load_confounds`](https://github.com/SIMEXP/load_confounds), we did try what you described here. I don't see why we should go back to the old approach, and make ICA-AROMA based strategy more error-prone. 
Would it be possible to have a lower level function that loads the specified confounds file? The high-level function could still parse the image file and call the lower-level function.

If [_load_single](https://github.com/nilearn/nilearn/blob/7d04e2fa49030784c6ec5132f1f831de7c6540fd/nilearn/interfaces/fmriprep/load_confounds.py#L293) and [_confounds_to_df](https://github.com/nilearn/nilearn/blob/7d04e2fa49030784c6ec5132f1f831de7c6540fd/nilearn/interfaces/fmriprep/load_confounds_utils.py#L200) were refactored such that confounds file is derived from the image file first, and then a separate function is called to actually load the confounds, that would produce a function that could load the confounds file without needing to specify an image file.

My use case is that I use ‘init_bold_confs_wf’ from fmriprep in my own workflow. So my confounds file is the same as fmriprep. I want to be able to load the confounds within the same workflow without having to datasink the outputs fmriprep style and then load the confounds in a separate workflow. 
As far as I am concerned, the aim is to work with the canonical version of fMRIPrep only.

If we refactor it in the way you describe, I would prefer to keep that lower level function private.
That should allow your use case, while maintaining the API supporting canonical fMRIPrep version for general user.
Does this sounds like a good idea to you? 

I will wait for other devs to weigh in...
@tsalo I wonder what you think about supporting use cases that are tangentially derived from fMRIPrep / existing standard. 
That sounds good to me. Refactoring so that the functions are smaller has the additional benefit of making the code cleaner and more versatile.
@htwangtw I would be okay supporting some low-level loading functions for BIDS-compatible derivatives. We should check the current status of the functional derivatives BEP to make sure that confounds files are fairly stable, but if we want to start handling generic confounds files, I think that should go in the BIDS interface module. That said, it should really be basic stuff, since `load_confounds` is what we want users using, when applicable.
Fantastic!
I would break down the task into two parts:
1. @ellisdg would refactor `_load_single` and `_confounds_to_df` to allow loading confound files. We will achieve this by completely isolate the file loading from the `_load_single` function.

2. Move the file retrieval code to the dedicated BIDS interface.

My gut feeling is, completing the refactoring proposed by @ellisdg, it will make the code easier to isolate that part of the code into the BIDS interface module suggested by @tsalo. 


Resolving patch:
diff --git a/nilearn/interfaces/fmriprep/load_confounds.py b/nilearn/interfaces/fmriprep/load_confounds.py
--- a/nilearn/interfaces/fmriprep/load_confounds.py
+++ b/nilearn/interfaces/fmriprep/load_confounds.py
@@ -5,11 +5,15 @@
 """
 import warnings
 import pandas as pd
-from .load_confounds_utils import (_sanitize_confounds, _confounds_to_df,
-                                   _prepare_output, MissingConfound)
+from .load_confounds_utils import (_sanitize_confounds,
+                                   _prepare_output,
+                                   MissingConfound,
+                                   _get_confounds_file,
+                                   _load_confounds_file_as_dataframe,
+                                   _load_confounds_json,
+                                   _get_json)
 from . import load_confounds_components as components
 
-
 # Global variables listing the admissible types of noise components
 all_confounds = [
     "motion",
@@ -269,7 +273,7 @@ def load_confounds(img_files,
     confounds_out = []
     sample_mask_out = []
     for file in img_files:
-        sample_mask, conf = _load_single(
+        sample_mask, conf = _load_confounds_for_single_image_file(
             file, strategy, demean,
             motion=motion,
             scrub=scrub,
@@ -290,34 +294,57 @@ def load_confounds(img_files,
     return confounds_out, sample_mask_out
 
 
-def _load_single(confounds_raw, strategy, demean, **kargs):
+def _load_confounds_for_single_image_file(image_file, strategy, demean,
+                                          **kwargs):
     """Load confounds for a single image file."""
-    # Convert tsv file to pandas dataframe
-    # check if relevant imaging files are present according to the strategy
-    flag_acompcor = ("compcor" in strategy) and (
-        "anat" in kargs.get("compcor")
-    )
+    # Check for ica_aroma in strategy, this will change the required image_file
     flag_full_aroma = ("ica_aroma" in strategy) and (
-        kargs.get("ica_aroma") == "full"
+        kwargs.get("ica_aroma") == "full"
     )
-    confounds_raw, meta_json = _confounds_to_df(
-        confounds_raw, flag_acompcor, flag_full_aroma
+
+    confounds_file = _get_confounds_file(image_file,
+                                         flag_full_aroma=flag_full_aroma)
+    confounds_json_file = _get_json(confounds_file)
+
+    return _load_single_confounds_file(confounds_file=confounds_file,
+                                       strategy=strategy,
+                                       demean=demean,
+                                       confounds_json_file=confounds_json_file,
+                                       **kwargs)
+
+
+def _load_single_confounds_file(confounds_file, strategy, demean=True,
+                                confounds_json_file=None, **kwargs):
+    """Load and extract specified confounds from the confounds file."""
+    flag_acompcor = ("compcor" in strategy) and (
+        "anat" in kwargs.get("compcor")
     )
+    # Convert tsv file to pandas dataframe
+    confounds_all = _load_confounds_file_as_dataframe(confounds_file)
+
+    if confounds_json_file is None:
+        confounds_json_file = _get_json(confounds_file)
+
+    # Read the associated json file
+    meta_json = _load_confounds_json(confounds_json_file,
+                                     flag_acompcor=flag_acompcor)
 
     missing = {"confounds": [], "keywords": []}
     # always check non steady state volumes are loaded
-    confounds, missing = _load_noise_component(confounds_raw,
-                                               "non_steady_state",
-                                               missing, meta_json=meta_json,
-                                               **kargs)
+    confounds_select, missing = _load_noise_component(confounds_all,
+                                                      "non_steady_state",
+                                                      missing,
+                                                      meta_json=meta_json,
+                                                      **kwargs)
     for component in strategy:
         loaded_confounds, missing = _load_noise_component(
-            confounds_raw, component, missing, meta_json=meta_json,
-            **kargs)
-        confounds = pd.concat([confounds, loaded_confounds], axis=1)
+            confounds_all, component, missing, meta_json=meta_json,
+            **kwargs)
+        confounds_select = pd.concat([confounds_select, loaded_confounds],
+                                     axis=1)
 
     _check_error(missing)  # raise any missing
-    return _prepare_output(confounds, demean)
+    return _prepare_output(confounds_select, demean)
 
 
 def _load_noise_component(confounds_raw, component, missing, **kargs):
diff --git a/nilearn/interfaces/fmriprep/load_confounds_utils.py b/nilearn/interfaces/fmriprep/load_confounds_utils.py
--- a/nilearn/interfaces/fmriprep/load_confounds_utils.py
+++ b/nilearn/interfaces/fmriprep/load_confounds_utils.py
@@ -117,10 +117,20 @@ def _get_file_name(nii_file):
         return confounds_raw[0]
 
 
-def _get_json(confounds_raw_path, flag_acompcor):
-    """Load json data companion to the confounds tsv file."""
+def _get_confounds_file(image_file, flag_full_aroma):
+    _check_images(image_file, flag_full_aroma)
+    confounds_raw_path = _get_file_name(image_file)
+    return confounds_raw_path
+
+
+def _get_json(confounds_raw_path):
+    """return json data companion file to the confounds tsv file."""
     # Load JSON file
-    confounds_json = confounds_raw_path.replace("tsv", "json")
+    return confounds_raw_path.replace("tsv", "json")
+
+
+def _load_confounds_json(confounds_json, flag_acompcor):
+    """Load json data companion to the confounds tsv file."""
     try:
         with open(confounds_json, "rb") as f:
             confounds_json = json.load(f)
@@ -137,7 +147,7 @@ def _get_json(confounds_raw_path, flag_acompcor):
     return confounds_json
 
 
-def _get_file_raw(confounds_raw_path):
+def _load_confounds_file_as_dataframe(confounds_raw_path):
     """Load raw confounds as a pandas DataFrame."""
     confounds_raw = pd.read_csv(
         confounds_raw_path, delimiter="\t", encoding="utf-8"
@@ -197,15 +207,6 @@ def _check_images(image_file, flag_full_aroma):
         raise ValueError(error_message)
 
 
-def _confounds_to_df(image_file, flag_acompcor, flag_full_aroma):
-    """Load raw confounds and associated metadata."""
-    _check_images(image_file, flag_full_aroma)
-    confounds_raw_path = _get_file_name(image_file)
-    confounds_json = _get_json(confounds_raw_path, flag_acompcor)
-    confounds_raw = _get_file_raw(confounds_raw_path)
-    return confounds_raw, confounds_json
-
-
 def _prepare_output(confounds, demean):
     """Demean and create sample mask for the selected confounds."""
     sample_mask, confounds, outliers = _extract_outlier_regressors(confounds)


Test patch:
diff --git a/nilearn/interfaces/fmriprep/tests/test_load_confounds.py b/nilearn/interfaces/fmriprep/tests/test_load_confounds.py
--- a/nilearn/interfaces/fmriprep/tests/test_load_confounds.py
+++ b/nilearn/interfaces/fmriprep/tests/test_load_confounds.py
@@ -8,7 +8,8 @@
 from nibabel import Nifti1Image
 from nilearn.maskers import NiftiMasker
 from nilearn.interfaces.fmriprep import load_confounds
-from nilearn.interfaces.fmriprep.load_confounds import _check_strategy
+from nilearn.interfaces.fmriprep.load_confounds import _check_strategy, \
+    _load_single_confounds_file
 
 from nilearn._utils.fmriprep_confounds import _to_camel_case
 
@@ -242,6 +243,28 @@ def test_confounds2df(tmp_path):
     assert "trans_x" in confounds.columns
 
 
+def test_load_single_confounds_file(tmp_path):
+    """Check that the load_confounds function returns the same confounds as
+    _load_single_confounds_file."""
+    nii_file, confounds_file = create_tmp_filepath(tmp_path,
+                                                   copy_confounds=True)
+
+    # get defaults from load_confounds
+    import inspect
+    _defaults = {key: value.default for key, value
+                 in inspect.signature(load_confounds).parameters.items()}
+    _defaults.pop("img_files")
+    _default_strategy = _defaults.pop("strategy")
+
+    _, confounds = _load_single_confounds_file(str(confounds_file),
+                                               strategy=_default_strategy,
+                                               **_defaults)
+    confounds_nii, _ = load_confounds(nii_file,
+                                      strategy=_default_strategy,
+                                      **_defaults)
+    pd.testing.assert_frame_equal(confounds, confounds_nii)
+
+
 @pytest.mark.parametrize("strategy,message",
                          [(["string", ], "not a supported type of confounds."),
                           ("error", "tuple or list of strings"),