Repository: nilearn/nilearn

Issue:
`fetch_openneuro_dataset_index` should clearly indicate it's hardcoded to retrieve a particular dataset
The docstring for `nilearn.datasets.fetch_openneuro_dataset_index` implies the function can be used to retrieve the file index from any OpenNeuro dataset, but it's actually hardcoded to only work with one dataset (ds000030). This is confusing, as the local directory takes the name of the requested dataset, even though it's always the same file list that's retrieved. A generalized function would be best if feasible, but failing that, it probably makes sense to only take a `version` argument and clarify in the docstring that the scope of the utility is limited to ds000030.


Discussion:
Thanks for opening this @tyarkoni. I agree that this can be confusing as the URL is indeed harcoded and doesn't depend on the value of the `dataset_version` argument:

https://github.com/nilearn/nilearn/blob/7142c5c9199e13c67111e7a44c58c9169259908b/nilearn/datasets/func.py#L2295-L2302
 
I think there are two things we can do quickly to improve this:

- Clarify the docstring to reflect the fact that the argument `dataset_version` cannot take any value (currently only one).
- Add a safety check that raises an error for invalid values of this argument (which would avoid creating data folders with any string you pass in).


`fetch_openneuro_dataset` also seems to rely on `fetch_openneuro_dataset_index` and work only for that particular dataset.
I think they should be called something like `fetch_ucla_neuropsychiatric_phenomics[_index]` or some other summary of the dataset name, and not `fetch_openneuro`

also should `fetch_openneuro_dataset_index` and `patch_openneuro_dataset` be private?
>also should fetch_openneuro_dataset_index and patch_openneuro_dataset be private?

I also think they should. Note that `fetch_openneuro_dataset_index` is used in an example but I don't really see why it is called and not directly `fetch_openneuro_dataset` (might just be to link this function to an example in the doc...).
ah yes it is to filter the urls to download with `fetch_openneuro_dataset` using `select_from_index` so I guess this one has to be public.
Ultimately, it seems like `fetch_openneuro_dataset_index` and `fetch_openneuro_dataset` should be merged in some way. However, as a more immediate solution, what if we (1) renamed the functions to something like `fetch_ds000030`, (2) made the current function names aliases, and (3) added a deprecation warning to the aliases?

EDIT: This would also help address https://github.com/nilearn/nilearn/issues/3128#issuecomment-1010376432.

Resolving patch:
diff --git a/examples/04_glm_first_level/plot_bids_features.py b/examples/04_glm_first_level/plot_bids_features.py
--- a/examples/04_glm_first_level/plot_bids_features.py
+++ b/examples/04_glm_first_level/plot_bids_features.py
@@ -36,10 +36,13 @@
 # This dataset contains the necessary information to run a statistical analysis
 # using Nilearn. The dataset also contains statistical results from a previous
 # FSL analysis that we can employ for comparison with the Nilearn estimation.
-from nilearn.datasets import (fetch_openneuro_dataset_index,
-                              fetch_openneuro_dataset, select_from_index)
+from nilearn.datasets import (
+    fetch_ds000030_urls,
+    fetch_openneuro_dataset,
+    select_from_index,
+)
 
-_, urls = fetch_openneuro_dataset_index()
+_, urls = fetch_ds000030_urls()
 
 exclusion_patterns = ['*group*', '*phenotype*', '*mriqc*',
                       '*parameter_plots*', '*physio_plots*',
diff --git a/nilearn/datasets/__init__.py b/nilearn/datasets/__init__.py
--- a/nilearn/datasets/__init__.py
+++ b/nilearn/datasets/__init__.py
@@ -18,6 +18,7 @@
                    fetch_surf_nki_enhanced, fetch_development_fmri,
                    fetch_language_localizer_demo_dataset,
                    fetch_bids_langloc_dataset,
+                   fetch_ds000030_urls,
                    fetch_openneuro_dataset_index,
                    select_from_index,
                    patch_openneuro_dataset,
diff --git a/nilearn/datasets/func.py b/nilearn/datasets/func.py
--- a/nilearn/datasets/func.py
+++ b/nilearn/datasets/func.py
@@ -1779,49 +1779,114 @@ def fetch_bids_langloc_dataset(data_dir=None, verbose=1):
 
 
 @fill_doc
-def fetch_openneuro_dataset_index(data_dir=None,
-                                  dataset_version='ds000030_R1.0.4',
-                                  verbose=1):
+def fetch_openneuro_dataset_index(
+    data_dir=None,
+    dataset_version='ds000030_R1.0.4',
+    verbose=1,
+):
     """Download a file with OpenNeuro :term:`BIDS` dataset index.
 
+    .. deprecated:: 0.9.2
+        `fetch_openneuro_dataset_index` will be removed in 0.11.
+
     Downloading the index allows to explore the dataset directories
     to select specific files to download. The index is a sorted list of urls.
 
     Parameters
     ----------
     %(data_dir)s
-    dataset_version : string, optional
+    dataset_version : :obj:`str`, optional
         Dataset version name. Assumes it is of the form [name]_[version].
         Default='ds000030_R1.0.4'.
+
+        .. warning:: Any value other than the default will be ignored.
+
+    %(verbose)s'
+
+    Returns
+    -------
+    urls_path : :obj:`str`
+        Path to downloaded dataset index.
+    urls : :obj:`list` of :obj:`str`
+        Sorted list of dataset directories.
+    """
+    warnings.warn(
+        (
+            'The "fetch_openneuro_dataset_index" function was deprecated in '
+            'version 0.9.2, and will be removed in 0.11. '
+            'Please use "fetch_ds000030_urls" instead.'
+        ),
+        DeprecationWarning,
+    )
+
+    DATASET_VERSION = 'ds000030_R1.0.4'
+    if dataset_version != DATASET_VERSION:
+        warnings.warn(
+            (
+                'An improper dataset_version has been provided. '
+                '"ds000030_R1.0.4" will be downloaded.'
+            ),
+            UserWarning,
+        )
+
+    urls_path, urls = fetch_ds000030_urls(data_dir=data_dir, verbose=verbose)
+    return urls_path, urls
+
+
+@fill_doc
+def fetch_ds000030_urls(data_dir=None, verbose=1):
+    """Fetch URLs for files from the ds000030 :term:`BIDS` dataset.
+
+    .. versionadded:: 0.9.2
+
+    This dataset is version 1.0.4 of the "UCLA Consortium for
+    Neuropsychiatric Phenomics LA5c" dataset
+    :footcite:p:`poldrack_phenome-wide_2016`.
+
+    Downloading the index allows users to explore the dataset directories
+    to select specific files to download.
+    The index is a sorted list of urls.
+
+    Parameters
+    ----------
+    %(data_dir)s
     %(verbose)s
 
     Returns
     -------
-    urls_path : string
+    urls_path : :obj:`str`
         Path to downloaded dataset index.
 
-    urls : list of string
+    urls : :obj:`list` of :obj:`str`
         Sorted list of dataset directories.
 
+    References
+    ----------
+    .. footbibliography::
     """
-    data_prefix = '{}/{}/uncompressed'.format(dataset_version.split('_')[0],
-                                              dataset_version,
-                                              )
-    data_dir = _get_dataset_dir(data_prefix, data_dir=data_dir,
-                                verbose=verbose)
+    DATA_PREFIX = 'ds000030/ds000030_R1.0.4/uncompressed'
+    FILE_URL = 'https://osf.io/86xj7/download'
+
+    data_dir = _get_dataset_dir(
+        DATA_PREFIX,
+        data_dir=data_dir,
+        verbose=verbose,
+    )
 
-    file_url = 'https://osf.io/86xj7/download'
     final_download_path = os.path.join(data_dir, 'urls.json')
-    downloaded_file_path = _fetch_files(data_dir=data_dir,
-                                        files=[(final_download_path,
-                                                file_url,
-                                                {'move': final_download_path}
-                                                )],
-                                        resume=True
-                                        )
+    downloaded_file_path = _fetch_files(
+        data_dir=data_dir,
+        files=[(
+            final_download_path,
+            FILE_URL,
+            {'move': final_download_path},
+        )],
+        resume=True
+    )
     urls_path = downloaded_file_path[0]
     with open(urls_path, 'r') as json_file:
         urls = json.load(json_file)
+
     return urls_path, urls
 
 
@@ -1894,45 +1959,68 @@ def infer_subjects(urls):
 
 
 def patch_openneuro_dataset(file_list):
-    """Add symlinks for files not named according to latest :term:`BIDS` conventions.
+    """Add symlinks for files not named according to :term:`BIDS` conventions.
+
+    .. warning::
+        This function uses a series of hardcoded patterns to generate the
+        corrected filenames.
+        These patterns are not comprehensive and this function is not
+        guaranteed to produce BIDS-compliant files.
+
+    Parameters
+    ----------
+    file_list : :obj:`list` of :obj:`str`
+        A list of filenames to update.
     """
-    rep = {'_T1w_brainmask': '_desc-brain_mask',
-           '_T1w_preproc': '_desc-preproc_T1w',
-           '_T1w_space-MNI152NLin2009cAsym_brainmask':
-               '_space-MNI152NLin2009cAsym_desc-brain_mask',
-           '_T1w_space-MNI152NLin2009cAsym_class-':
-               '_space-MNI152NLin2009cAsym_label-',
-           '_T1w_space-MNI152NLin2009cAsym_preproc':
-               '_space-MNI152NLin2009cAsym_desc-preproc_T1w',
-           '_bold_confounds': '_desc-confounds_regressors',
-           '_bold_space-MNI152NLin2009cAsym_brainmask':
-               '_space-MNI152NLin2009cAsym_desc-brain_mask',
-           '_bold_space-MNI152NLin2009cAsym_preproc':
-               '_space-MNI152NLin2009cAsym_desc-preproc_bold'
-           }
+    REPLACEMENTS = {
+        '_T1w_brainmask': '_desc-brain_mask',
+        '_T1w_preproc': '_desc-preproc_T1w',
+        '_T1w_space-MNI152NLin2009cAsym_brainmask':
+            '_space-MNI152NLin2009cAsym_desc-brain_mask',
+        '_T1w_space-MNI152NLin2009cAsym_class-':
+            '_space-MNI152NLin2009cAsym_label-',
+        '_T1w_space-MNI152NLin2009cAsym_preproc':
+            '_space-MNI152NLin2009cAsym_desc-preproc_T1w',
+        '_bold_confounds': '_desc-confounds_regressors',
+        '_bold_space-MNI152NLin2009cAsym_brainmask':
+            '_space-MNI152NLin2009cAsym_desc-brain_mask',
+        '_bold_space-MNI152NLin2009cAsym_preproc':
+            '_space-MNI152NLin2009cAsym_desc-preproc_bold'
+    }
+
     # Create a symlink if a file with the modified filename does not exist
-    for old in rep:
+    for old_pattern, new_pattern in REPLACEMENTS.items():
         for name in file_list:
-            if old in name:
-                if not os.path.exists(name.replace(old, rep[old])):
-                    os.symlink(name, name.replace(old, rep[old]))
+            if old_pattern in name:
+                new_name = name.replace(old_pattern, new_pattern)
+                if not os.path.exists(new_name):
+                    os.symlink(name, new_name)
 
 
 @fill_doc
 def fetch_openneuro_dataset(
-    urls=None, data_dir=None, dataset_version='ds000030_R1.0.4',
-    verbose=1):
+    urls=None,
+    data_dir=None,
+    dataset_version='ds000030_R1.0.4',
+    verbose=1,
+):
     """Download OpenNeuro :term:`BIDS` dataset.
 
+    This function specifically downloads files from a series of URLs.
+    Unless you use :func:`fetch_ds000030_urls` or the default parameters,
+    it is up to the user to ensure that the URLs are correct,
+    and that they are associated with an OpenNeuro dataset.
+
     Parameters
     ----------
     urls : list of string, optional
-        Openneuro url list of dataset files to download. If not specified
-        all files of the specified dataset will be downloaded.
+        List of URLs to dataset files to download.
+        If not specified, all files from the default dataset
+        (``ds000030_R1.0.4``) will be downloaded.
     %(data_dir)s
     dataset_version : string, optional
         Dataset version name. Assumes it is of the form [name]_[version].
-        Default is `ds000030_R1.0.4`.
+        Default is ``ds000030_R1.0.4``.
     %(verbose)s
 
     Returns
@@ -1943,21 +2031,74 @@ def fetch_openneuro_dataset(
     downloaded_files : list of string
         Absolute paths of downloaded files on disk.
 
-    """
-    data_prefix = '{}/{}/uncompressed'.format(
-        dataset_version.split('_')[0], dataset_version)
-    data_dir = _get_dataset_dir(data_prefix, data_dir=data_dir,
-                                verbose=verbose)
+    Notes
+    -----
+    The default dataset downloaded by this function is the
+    "UCLA Consortium for Neuropsychiatric Phenomics LA5c" dataset
+    :footcite:p:`poldrack_phenome-wide_2016`.
+
+    This copy includes filenames that are not compliant with the current
+    version of :term:`BIDS`, so this function also calls
+    :func:`patch_openneuro_dataset` to generate BIDS-compliant symlinks.
 
+    See Also
+    --------
+    :func:`fetch_ds000030_urls`
+    :func:`patch_openneuro_dataset`
+
+    References
+    ----------
+    .. footbibliography::
+    """
     # if urls are not specified we download the complete dataset index
     if urls is None:
-        _, urls = fetch_openneuro_dataset_index(
-            data_dir=data_dir, dataset_version=dataset_version,
-            verbose=verbose)
+        DATASET_VERSION = 'ds000030_R1.0.4'
+        if dataset_version != DATASET_VERSION:
+            warnings.warn(
+                'If `dataset_version` is not "ds000030_R1.0.4", '
+                '`urls` must be specified. Downloading "ds000030_R1.0.4".'
+            )
+
+        data_prefix = '{}/{}/uncompressed'.format(
+            DATASET_VERSION.split('_')[0],
+            DATASET_VERSION,
+        )
+        orig_data_dir = data_dir
+        data_dir = _get_dataset_dir(
+            data_prefix,
+            data_dir=data_dir,
+            verbose=verbose,
+        )
+
+        _, urls = fetch_ds000030_urls(
+            data_dir=orig_data_dir,
+            verbose=verbose,
+        )
+    else:
+        data_prefix = '{}/{}/uncompressed'.format(
+            dataset_version.split('_')[0],
+            dataset_version,
+        )
+        data_dir = _get_dataset_dir(
+            data_prefix,
+            data_dir=data_dir,
+            verbose=verbose,
+        )
 
     # The files_spec needed for _fetch_files
     files_spec = []
     files_dir = []
+
+    # Check that data prefix is found in each URL
+    bad_urls = [url for url in urls if data_prefix not in url]
+    if bad_urls:
+        raise ValueError(
+            f'data_prefix ({data_prefix}) is not found in at least one URL. '
+            'This indicates that the URLs do not correspond to the '
+            'dataset_version provided.\n'
+            f'Affected URLs: {bad_urls}'
+        )
+
     for url in urls:
         url_path = url.split(data_prefix + '/')[1]
         file_dir = os.path.join(data_dir, url_path)
@@ -1974,13 +2115,19 @@ def fetch_openneuro_dataset(
         while download_attempts > 0 and not success:
             try:
                 downloaded_files = _fetch_files(
-                    file_dir, [file_spec], resume=True, verbose=verbose)
+                    file_dir,
+                    [file_spec],
+                    resume=True,
+                    verbose=verbose,
+                )
                 downloaded += downloaded_files
                 success = True
             except Exception:
                 download_attempts -= 1
+
         if not success:
-            raise Exception('multiple failures downloading %s' % file_spec[1])
+            raise Exception(f'multiple failures downloading {file_spec[1]}')
+
     patch_openneuro_dataset(downloaded)
 
     return data_dir, sorted(downloaded)


Test patch:
diff --git a/nilearn/datasets/tests/test_func.py b/nilearn/datasets/tests/test_func.py
--- a/nilearn/datasets/tests/test_func.py
+++ b/nilearn/datasets/tests/test_func.py
@@ -688,7 +688,7 @@ def test_select_from_index(request_mocker):
     assert data_prefix + '/sub-xxx/ses-02_task-rest.txt' in new_urls
 
 
-def test_fetch_openneuro_dataset_index(request_mocker):
+def test_fetch_ds000030_urls(request_mocker):
     with TemporaryDirectory() as tmpdir:
         dataset_version = 'ds000030_R1.0.4'
         subdir_names = ['ds000030', 'ds000030_R1.0.4', 'uncompressed']
@@ -702,35 +702,70 @@ def test_fetch_openneuro_dataset_index(request_mocker):
         mock_json_content = ['junk1', 'junk2']
         with open(filepath, 'w') as f:
             json.dump(mock_json_content, f)
-        urls_path, urls = func.fetch_openneuro_dataset_index(
+
+        # fetch_ds000030_urls should retrieve the appropriate URLs
+        urls_path, urls = func.fetch_ds000030_urls(
             data_dir=tmpdir,
-            dataset_version=dataset_version,
             verbose=1,
         )
         urls_path = urls_path.replace('/', os.sep)
         assert urls_path == filepath
         assert urls == mock_json_content
 
+        # fetch_openneuro_dataset_index should do the same, but with a warning
+        with pytest.warns(DeprecationWarning):
+            urls_path, urls = func.fetch_openneuro_dataset_index(
+                data_dir=tmpdir,
+                dataset_version=dataset_version,
+                verbose=1,
+            )
+
+        urls_path = urls_path.replace('/', os.sep)
+        assert urls_path == filepath
+        assert urls == mock_json_content
+
+        # fetch_openneuro_dataset_index should even grab ds000030 when you
+        # provide a different dataset name
+        with pytest.warns(
+            UserWarning,
+            match='"ds000030_R1.0.4" will be downloaded',
+        ):
+            urls_path, urls = func.fetch_openneuro_dataset_index(
+                data_dir=tmpdir,
+                dataset_version='ds500_v2',
+                verbose=1,
+            )
+
+        urls_path = urls_path.replace('/', os.sep)
+        assert urls_path == filepath
+        assert urls == mock_json_content
+
 
 def test_fetch_openneuro_dataset(request_mocker, tmp_path):
     dataset_version = 'ds000030_R1.0.4'
     data_prefix = '{}/{}/uncompressed'.format(
-        dataset_version.split('_')[0], dataset_version)
-    data_dir = _get_dataset_dir(data_prefix, data_dir=tmp_path,
-                                verbose=1)
+        dataset_version.split('_')[0],
+        dataset_version,
+    )
+    data_dir = _get_dataset_dir(
+        data_prefix,
+        data_dir=tmp_path,
+        verbose=1,
+    )
     url_file = os.path.join(data_dir, 'urls.json')
+
     # Prepare url files for subject and filter tests
     urls = [
-        "https://example.com/{}/stuff.html" + data_prefix + '',
-        "https://example.com/{}/sub-xxx.html",
-        "https://example.com/{}/sub-yyy.html",
-        "https://example.com/{}/sub-xxx/ses-01_task-rest.txt",
-        "https://example.com/{}/sub-xxx/ses-01_task-other.txt",
-        "https://example.com/{}/sub-xxx/ses-02_task-rest.txt",
-        "https://example.com/{}/sub-xxx/ses-02_task-other.txt",
-        "https://example.com/{}/sub-yyy/ses-01.txt",
-        "https://example.com/{}/sub-yyy/ses-02.txt"]
-    urls = [url.format(data_prefix) for url in urls]
+        f'https://example.com/{data_prefix}/stuff.html',
+        f'https://example.com/{data_prefix}/sub-xxx.html',
+        f'https://example.com/{data_prefix}/sub-yyy.html',
+        f'https://example.com/{data_prefix}/sub-xxx/ses-01_task-rest.txt',
+        f'https://example.com/{data_prefix}/sub-xxx/ses-01_task-other.txt',
+        f'https://example.com/{data_prefix}/sub-xxx/ses-02_task-rest.txt',
+        f'https://example.com/{data_prefix}/sub-xxx/ses-02_task-other.txt',
+        f'https://example.com/{data_prefix}/sub-yyy/ses-01.txt',
+        f'https://example.com/{data_prefix}/sub-yyy/ses-02.txt',
+    ]
     json.dump(urls, open(url_file, 'w'))
 
     # Only 1 subject and not subject specific files get downloaded
@@ -740,6 +775,27 @@ def test_fetch_openneuro_dataset(request_mocker, tmp_path):
     assert isinstance(dl_files, list)
     assert len(dl_files) == 9
 
+    # URLs do not contain the data_prefix, which should raise a ValueError
+    urls = [
+        'https://example.com/stuff.html',
+        'https://example.com/sub-yyy/ses-01.txt',
+    ]
+    with pytest.raises(ValueError, match='This indicates that the URLs'):
+        func.fetch_openneuro_dataset(urls, tmp_path, dataset_version)
+
+    # Try downloading a different dataset without providing URLs
+    # This should raise a warning and download ds000030.
+    with pytest.warns(
+        UserWarning,
+        match='Downloading "ds000030_R1.0.4".',
+    ):
+        urls_path, urls = func.fetch_openneuro_dataset(
+            urls=None,
+            data_dir=tmp_path,
+            dataset_version='ds500_v2',
+            verbose=1,
+        )
+
 
 def test_fetch_localizer(request_mocker, tmp_path):
     dataset = func.fetch_localizer_first_level(data_dir=tmp_path)