Repository: mne-tools/mne-python

Issue:
Sample data set cannot be downloaded if data folder does not exist
The default location for MNE sample data is `~/mne_data`. Apparently, if this folder does not exist the data set cannot be downloaded:

```python
# make sure the folder "~/mne_data" does not exist!
import mne
from mne.datasets import sample


data_path = sample.data_path()
raw_fname = data_path + '/MEG/sample/sample_audvis_filt-0-40_raw.fif'

raw = mne.io.read_raw_fif(raw_fname)
```

I get the following error:

```
Downloading archive MNE-sample-data-processed.tar.gz to /Users/clemens/mne_data
Downloading https://files.osf.io/v1/resources/rxvq7/providers/osfstorage/59c0e26f9ad5a1025c4ab159?version=5&action=download&direct (1.54 GB)
  0%|          | Downloading : 0.00/1.54G [00:00<?,        ?B/s]Error while fetching file https://osf.io/86qa2/download?version=5. Dataset fetching aborted.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3418, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-9a4a5c6c7639>", line 1, in <module>
    runfile('/Users/clemens/Library/Application Support/JetBrains/PyCharm2020.3/scratches/scratch.py', wdir='/Users/clemens/Library/Application Support/JetBrains/PyCharm2020.3/scratches')
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/Users/clemens/Library/Application Support/JetBrains/PyCharm2020.3/scratches/scratch.py", line 5, in <module>
    data_path = sample.data_path()
  File "<decorator-gen-461>", line 24, in data_path
  File "/Users/clemens/Repositories/mne-python/mne/datasets/sample/sample.py", line 19, in data_path
    return _data_path(path=path, force_update=force_update,
  File "/Users/clemens/Repositories/mne-python/mne/datasets/utils.py", line 418, in _data_path
    remove_archive, full = _download(path, u, an, h)
  File "/Users/clemens/Repositories/mne-python/mne/datasets/utils.py", line 475, in _download
    _fetch_file(url, full_name, print_destination=False,
  File "<decorator-gen-3>", line 24, in _fetch_file
  File "/Users/clemens/Repositories/mne-python/mne/utils/fetching.py", line 117, in _fetch_file
    _get_http(url, temp_file_name, initial_size, timeout, verbose_bool)
  File "/Users/clemens/Repositories/mne-python/mne/utils/fetching.py", line 55, in _get_http
    with open(temp_file_name, mode) as local_file:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/clemens/mne_data/MNE-sample-data-processed.tar.gz.part'
```

Everything works as expected if the folder `~/mne_data` exists (even if it is just an empty folder). We should automatically create this folder if not present to prevent this error.


Discussion:
that's weird. It works great on collab for example. I use it all the time
with students

>

@cbrn Maybe this folder was created and then subsequently deleted? I remember I had a similar problem in such situation - the mne config was pointing to non-existent directory.
@mmagnuski exactly!
We could consider using something like `os.makedirs` to deal with this use case but the safer solution I think is to check `if not op.isdir(path)` and if so `raise FileNotFoundError` so that people can create the directory themselves if they really want it to be at that path.
I think it's ok/preferable to raise an error if the folder is specified in the config but missing on disk. I'm thinking of cases where the data is on a network drive that is unavailable; we don't want to automatically create a folder at its mountpoint and download data to it... When the network drive is restored that data will be masked but still using disk space on the local drive
I think automatically creating the folder is less confusing. After all, that's what happens if there is no config file yet. Now if someone has already created a config file but removed the folder for some reason, I'd expect the download to still work (and go into the folder I've specified).
> After all, that's what happens if there is no config file yet.

to me this is a key difference.  If config says "look here" and the folder is missing, we know something is wrong because when the config value was initially set we would have created it. So raising an error when something is wrong seems OK.  I'd also like to reiterate what I said above about network drives: (1) auto-recreating the folder risks wasting time and bandwidth in cases where a user simply forgot to re-mount their network drive, and (2) for users working on shared servers with strong disk quotas, automatically creating a folder and immediately downloading a large dataset to it can easily fail in other ways.

I have a few compromise suggestions:

1. continue to raise a `FileNotFound` error but make the message more informative, like "config said to store datasets here, but the folder doesn't exist, please create it manually and re-run this command"
2. if the config-specified folder is missing, prompt the user "do you want me to create it?" and only error if they say no
3. add a param to all of the dataset download functions, like `create_if_missing` or so.
That's not transparent. When downloading a data set for the first time, MNE automatically creates the folder `mne_data` in the user's home (plus the config file). Now if a user decides to get rid of the folder (e.g. because it consumes too much disk space), but after a while decides to use some sample data set again, suddenly the download will error although there doesn't seem to be any visible difference (after all, `mne_data` didn't exist in the first place, and the hidden config folder `.mne` is likely not that obvious - but that's the only difference that's causing this behavior).

Also, at no point did the user explicitly specify to use the `~/mne_data` folder, that's just the default value.

I'm just saying if it tripped me up it's likely to confuse new users as well. Your compromise suggestions are fine, but IMO they are overkill. I'd just create the folder and not let that depend on whether or not a `.mne` config folder has been created.
I think that given we don't know what caused the config directory to be unavailable it might be safer to throw a very informative error and ask the user to check what is wrong (and recreate the directory if that is what they want). This seems to be safer and least overkill (on mne side) to me actually.
It's not the config directory that's missing, it's the data folder. I don't see why it should get created automatically in one case and throw an error in another.
> I don't see why it should get created automatically in one case and throw an error in another.

because in one case *there is no expectation of where to put it because the relevant config variable doesn't exist yet*.  Once the config variable exists, we have an expectation of where in the filesystem to put the files, and by extension we have an expectation that the folder should already exist (because if it didn't already exist we created it at the time we initted the config variable).

If the *only* option for locating the data folder were `~/mne_data` I might agree with you, but users can config that data folder to be anywhere in their filesystem and I'm very hesitant to go creating directories and filling them with MB or GB of data, at places that might be mountpoints for other drives.
> because in one case there is no expectation of where to put it because the relevant config variable doesn't exist yet. Once the config variable exists, we have an expectation of where in the filesystem to put the files, and by extension we have an expectation that the folder should already exist (because if it didn't already exist we created it at the time we initted the config variable).

Exactly - *we* have an expectation, the average user doesn't know what's going on.
I guess we won't agree on this one, so implementing your three points is at least better than the current behavior.
> It's not the config directory that's missing, it's the data folder.

Yes, sorry the "cofig directory" was short for "directory specified in the config file".

> I don't see why it should get created automatically in one case and throw an error in another.

The user could have moved that folder to another place / drive for example. In such case the config should be changed to point to the new location. The error message could explain that, while auto-downloading to the old location might be equally irritating to someone in this scenario as the current behavior is for you. :)
BTW, using e.g. `eegbci.load_data(1, 4)` instead of the sample data set works as I expected - it automatically creates the folder even if the folder specified in the config doesn't exist. I think that's how it should be for all datasets. So now I can argue that we are already inconsistent, but probably the solution is to add `load_data` methods to all datasets?

Resolving patch:
diff --git a/mne/datasets/utils.py b/mne/datasets/utils.py
--- a/mne/datasets/utils.py
+++ b/mne/datasets/utils.py
@@ -178,6 +178,11 @@ def _get_path(path, key, name):
     # 3. get_config('MNE_DATA')
     path = get_config(key, get_config('MNE_DATA'))
     if path is not None:
+        if not op.exists(path):
+            msg = (f"Download location {path} as specified by MNE_DATA does "
+                   f"not exist. Either create this directory manually and try "
+                   f"again, or set MNE_DATA to an existing directory.")
+            raise FileNotFoundError(msg)
         return path
     # 4. ~/mne_data (but use a fake home during testing so we don't
     #    unnecessarily create ~/mne_data)


Test patch:
diff --git a/mne/datasets/tests/test_datasets.py b/mne/datasets/tests/test_datasets.py
--- a/mne/datasets/tests/test_datasets.py
+++ b/mne/datasets/tests/test_datasets.py
@@ -13,7 +13,7 @@
 from mne.datasets._fsaverage.base import _set_montage_coreg_path
 from mne.datasets.utils import _manifest_check_download
 
-from mne.utils import (requires_good_network, modified_env,
+from mne.utils import (requires_good_network,
                        get_subjects_dir, ArgvSetter, _pl, use_log_level,
                        catch_logging, hashfunc)
 
@@ -21,7 +21,7 @@
 subjects_dir = op.join(testing.data_path(download=False), 'subjects')
 
 
-def test_datasets_basic(tmpdir):
+def test_datasets_basic(tmpdir, monkeypatch):
     """Test simple dataset functions."""
     # XXX 'hf_sef' and 'misc' do not conform to these standards
     for dname in ('sample', 'somato', 'spm_face', 'testing', 'opm',
@@ -45,13 +45,17 @@ def test_datasets_basic(tmpdir):
     tempdir = str(tmpdir)
     # don't let it read from the config file to get the directory,
     # force it to look for the default
-    with modified_env(**{'_MNE_FAKE_HOME_DIR': tempdir, 'SUBJECTS_DIR': None}):
-        assert (datasets.utils._get_path(None, 'foo', 'bar') ==
-                op.join(tempdir, 'mne_data'))
-        assert get_subjects_dir(None) is None
-        _set_montage_coreg_path()
-        sd = get_subjects_dir()
-        assert sd.endswith('MNE-fsaverage-data')
+    monkeypatch.setenv('_MNE_FAKE_HOME_DIR', tempdir)
+    monkeypatch.delenv('SUBJECTS_DIR', raising=False)
+    assert (datasets.utils._get_path(None, 'foo', 'bar') ==
+            op.join(tempdir, 'mne_data'))
+    assert get_subjects_dir(None) is None
+    _set_montage_coreg_path()
+    sd = get_subjects_dir()
+    assert sd.endswith('MNE-fsaverage-data')
+    monkeypatch.setenv('MNE_DATA', str(tmpdir.join('foo')))
+    with pytest.raises(FileNotFoundError, match='as specified by MNE_DAT'):
+        testing.data_path(download=False)
 
 
 @requires_good_network