Repository: scikit-image/scikit-image

Issue:
PermissionError when creating data_dir when user doesn't have permissions
## Description

<!--
(Note: for guidance on how to use `scikit-image`, please post instead on https://forum.image.sc/tag/scikit-image)
-->
Version 0.16.2 worked, but 0.17.1 tries to create a directory in whatever `appdirs` thinks is an appropriate cache directory.

For my flask app's service, the user is a no-login user, and somehow this resolves to `/var/www/`.
The user doesn't have permissions to create a directory there, so the flask app crashes with a PermissionError raised from skimage.

I specified `SKIMAGE_DATADIR=/tmp/.skimage_cache`, and ended up seeing #4660, as my flask app starts multiple processes.

## Way to reproduce
```python
import appdirs
# Monkey-patch appdirs.user_cache_dir to an unwritable location
appdirs.user_cache_dir = lambda _: '/var/log'
import skimage
```

```
Cannot create data cache folder '/var/log/0.17.1'. Will not be able to download remote data files. Use environment variable 'SKIMAGE_DATADIR' to specify another directory.
Traceback (most recent call last):
  File "skimage_issue.py", line 4, in <module>
    import skimage
  File "/home/vleduc/python/virtualenvs/py37/lib/python3.7/site-packages/skimage/__init__.py", line 133, in <module>
    from .data import data_dir
  File "/home/vleduc/python/virtualenvs/py37/lib/python3.7/site-packages/skimage/data/__init__.py", line 90, in <module>
    os.makedirs(data_dir, exist_ok=True)
  File "/home/vleduc/python/virtualenvs/py37/lib/python3.7/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/home/vleduc/python/virtualenvs/py37/lib/python3.7/os.py", line 223, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/var/log/0.17.1'
```


## Version information
```python
# Paste the output of the following python commands
from __future__ import print_function
import sys; print(sys.version)
import platform; print(platform.platform())
import skimage; print("scikit-image version: {}".format(skimage.__version__))
import numpy; print("numpy version: {}".format(numpy.__version__))
```

```python
3.7.5 (default, Nov  7 2019, 10:50:52) 
[GCC 8.3.0]
Linux-4.15.0-99-generic-x86_64-with-Ubuntu-18.04-bionic
scikit-image version: 0.17.1
numpy version: 1.18.4
```



Discussion:
Can you tell me if you start multiple processes before or after you import sxikit image?
@hmaarrfk it seems that skimage is trying to create the data dir at import time in https://github.com/scikit-image/scikit-image/blob/master/skimage/data/__init__.py#L90 Is that necessary? Could it be delayed to the `_fetch` function instead (with a check to see if it already exists)?

On the Pooch side, if we don't raise an exception with a permission error (only issue a warning as seen above). So it wouldn't cause any problems unless the user wants to load sample data.
Should we open an issue in appdirs regarding using `/tmp` if the home dir is not writable?

Opened one https://github.com/ActiveState/appdirs/issues/146
In a more generic way, tempfile.gettempdir()
This is definitely an issue we should workaround today. 

I'm just waiting until we merge in an other pull request relating to this new feature of ours.
@hmaarrfk Thanks for looking into it! I think multiple processes are started after skimage is imported. Although the problem occurs at import time.

Yeah I'm not sure if this should be an issue in appdirs, pooch or scikit-image. Although from my point of view as a user of scikit-image, I feel the library might be trying to do much for me without asking. The error states:

`Will not be able to download remote data files. Use environment variable 'SKIMAGE_DATADIR' to specify another directory.`

, but I don't even want to use this functionality. In my use case, I don't need it.
What about simply wrap the directory creation in a `try ... except ...` bloc?
@leducvin  you have been heard loud and clear, and we are working toward a solution that will work well with python 3.6 (though that isn't clear how we would do it).

Unfortunately, for now, this isn't an issue that we can solve right way.

We have a few contradictory requirements:
1. Shipping more examples to our users.
2. Enablign builds in applications such as your own.

For now, I would suggest you skip
- 0.17.1 (hard dependency on pooch)
- 0.17.2 (soft dependency on pooch, but if pooch is installed, the same problem will happen)

You may you use `!=0.17.1,!=0.17.2`

Rest assured that we are working to resolve this breaking change for your application.
No worries. I already started using `!=0.17.1`, and that is working fine. I'll update to `!=0.17.1,!=0.17.2`. Thanks for the heads up.

As mentioned in ActiveState/appdirs#146, setting a `$HOME` env variable to a writable dir should also be a workaround, although I haven't tried it at this time.
I agree that this should not cause failure. 

0.17.2 doesn't have the pooch requirement, but pooch may be installed by an other of your dependencies by accident.

We just need to balance a few other things to get it right. In my mind, you shouldn't have to use any configuration to get going in your usecase.

The only time you should set that environment variable is if you want to have a shared store of our future GB large datasets.

When using a AWS Lambda, still is also a problem. 
> @hmaarrfk Thanks for looking into it! I think multiple processes are started after skimage is imported. Although the problem occurs at import time.
> 
> Yeah I'm not sure if this should be an issue in appdirs, pooch or scikit-image. Although from my point of view as a user of scikit-image, I feel the library might be trying to do much for me without asking. The error states:
> 
> `Will not be able to download remote data files. Use environment variable 'SKIMAGE_DATADIR' to specify another directory.`
> 
> , but I don't even want to use this functionality. In my use case, I don't need it.

I was using this in my docker container. Setting up the environment variable solved my problem!

Thanks!
This issue has been mentioned on **Image.sc Forum**. There might be relevant details there:

https://forum.image.sc/t/why-does-init-py-attempt-to-download-sample-data/79665/4

Reproducable with `SKIMAGE_DATADIR=/etc/skimage python -c 'import skimage.data'`. Assuming that `/etc/skimage`) isn't writable this will produce give
```
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "~/Res/scikit-image/skimage/__init__.py", line 141, in <module>
from .data import data_dir
File "<frozen importlib._bootstrap>", line 1231, in _handle_fromlist
File "~/.local/lib/venv/skimagedev/lib/python3.11/site-packages/lazy_loader/__init__.py", line76, in __getattr__
submod = importlib.import_module(submod_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "~/.local/lib/micromamba/envs/calc/lib/python3.11/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "~/Res/scikit-image/skimage/data/_fetchers.py", line 263, in <module>
_init_pooch()
File "~/Res/scikit-image/skimage/data/_fetchers.py", line 240, in _init_pooch
os.makedirs(data_dir, exist_ok=True)
File "<frozen os>", line 215, in makedirs
File "<frozen os>", line 215, in makedirs
File "<frozen os>", line 225, in makedirs
PermissionError: [Errno 13] Permission denied: '/etc/skimage'
```
I'll look into fixing this.

Resolving patch:
diff --git a/skimage/data/_fetchers.py b/skimage/data/_fetchers.py
--- a/skimage/data/_fetchers.py
+++ b/skimage/data/_fetchers.py
@@ -9,15 +9,15 @@
 import shutil
 
 from ..util.dtype import img_as_bool
-from ._registry import registry, legacy_registry, registry_urls
+from ._registry import registry, registry_urls
 
 from .. import __version__
 
 import os.path as osp
 import os
 
-legacy_data_dir = osp.abspath(osp.dirname(__file__))
-skimage_distribution_dir = osp.join(legacy_data_dir, '..')
+_LEGACY_DATA_DIR = osp.dirname(__file__)
+_DISTRIBUTION_DIR = osp.dirname(_LEGACY_DATA_DIR)
 
 try:
     from pooch import file_hash
@@ -80,7 +80,7 @@ def create_image_fetcher():
     except ImportError:
         # Without pooch, fallback on the standard data directory
         # which for now, includes a few limited data samples
-        return None, legacy_data_dir
+        return None, _LEGACY_DATA_DIR
 
     # Pooch expects a `+` to exist in development versions.
     # Since scikit-image doesn't follow that convention, we have to manually
@@ -126,11 +126,6 @@ def create_image_fetcher():
 
 image_fetcher, data_dir = create_image_fetcher()
 
-if image_fetcher is None:
-    has_pooch = False
-else:
-    has_pooch = True
-
 
 def _skip_pytest_case_requiring_pooch(data_filename):
     """If a test case is calling pooch, skip it.
@@ -152,21 +147,39 @@ def _skip_pytest_case_requiring_pooch(data_filename):
                     allow_module_level=True)
 
 
+def _ensure_cache_dir(*, target_dir):
+    """Prepare local cache directory if it doesn't exist already.
+
+    Creates::
+
+        /path/to/target_dir/
+                 └─ data/
+                    └─ README.txt
+    """
+    os.makedirs(osp.join(target_dir, "data"), exist_ok=True)
+    readme_src = osp.join(_DISTRIBUTION_DIR, "data/README.txt")
+    readme_dest = osp.join(target_dir, "data/README.txt")
+    if not osp.exists(readme_dest):
+        shutil.copy2(readme_src, readme_dest)
+
+
 def _fetch(data_filename):
     """Fetch a given data file from either the local cache or the repository.
 
     This function provides the path location of the data file given
-    its name in the scikit-image repository.
+    its name in the scikit-image repository. If a data file is not included in the
+    distribution and pooch is available, it is downloaded and cached.
 
     Parameters
     ----------
-    data_filename:
+    data_filename : str
         Name of the file in the scikit-image repository. e.g.
         'restoration/tess/camera_rl.npy'.
 
     Returns
     -------
-    Path of the local file as a python string.
+    file_path : str
+        Path of the local file.
 
     Raises
     ------
@@ -181,33 +194,26 @@ def _fetch(data_filename):
         If scikit-image is unable to connect to the internet but the
         dataset has not been downloaded yet.
     """
-    resolved_path = osp.join(data_dir, '..', data_filename)
     expected_hash = registry[data_filename]
+    if image_fetcher is None:
+        cache_dir = osp.dirname(data_dir)
+    else:
+        cache_dir = str(image_fetcher.abspath)
 
-    # Case 1:
-    # The file may already be in the data_dir.
-    # We may have decided to ship it in the scikit-image distribution.
-    if _has_hash(resolved_path, expected_hash):
+    # Case 1: the file is already cached in `data_cache_dir`
+    cached_file_path = osp.join(cache_dir, data_filename)
+    if _has_hash(cached_file_path, expected_hash):
         # Nothing to be done, file is where it is expected to be
-        return resolved_path
-
-    # Case 2:
-    # The user is using a cloned version of the github repo, which
-    # contains both the publicly shipped data, and test data.
-    # In this case, the file would be located relative to the
-    # skimage_distribution_dir
-    gh_repository_path = osp.join(skimage_distribution_dir, data_filename)
-    if _has_hash(gh_repository_path, expected_hash):
-        parent = osp.dirname(resolved_path)
-        os.makedirs(parent, exist_ok=True)
-        shutil.copy2(gh_repository_path, resolved_path)
-        return resolved_path
-
-    # Case 3:
-    # Pooch not found.
+        return cached_file_path
+
+    # Case 2: file is present in `legacy_data_dir`
+    legacy_file_path = osp.join(_DISTRIBUTION_DIR, data_filename)
+    if _has_hash(legacy_file_path, expected_hash):
+        return legacy_file_path
+
+    # Case 3: file is not present locally
     if image_fetcher is None:
         _skip_pytest_case_requiring_pooch(data_filename)
-
         raise ModuleNotFoundError(
             "The requested file is part of the scikit-image distribution, "
             "but requires the installation of an optional dependency, pooch. "
@@ -215,16 +221,13 @@ def _fetch(data_filename):
             "Follow installation instruction found at "
             "https://scikit-image.org/docs/stable/install.html"
         )
-
-    # Case 4:
-    # Pooch needs to download the data. Let the image fetcher to search for
-    # our data. A ConnectionError is raised if no internet connection is
-    # available.
+    # Download the data with pooch which caches it automatically
+    _ensure_cache_dir(target_dir=cache_dir)
     try:
-        resolved_path = image_fetcher.fetch(data_filename)
+        cached_file_path = image_fetcher.fetch(data_filename)
+        return cached_file_path
     except ConnectionError as err:
         _skip_pytest_case_requiring_pooch(data_filename)
-
         # If we decide in the future to suppress the underlying 'requests'
         # error, change this to `raise ... from None`. See PEP 3134.
         raise ConnectionError(
@@ -233,34 +236,6 @@ def _fetch(data_filename):
             'future, try `skimage.data.download_all()` when you are '
             'connected to the internet.'
         ) from err
-    return resolved_path
-
-
-def _init_pooch():
-    os.makedirs(data_dir, exist_ok=True)
-
-    # Copy in the README.txt if it doesn't already exist.
-    # If the file was originally copied to the data cache directory read-only
-    # then we cannot overwrite it, nor do we need to copy on every init.
-    # In general, as the data cache directory contains the scikit-image version
-    # it should not be necessary to overwrite this file as it should not
-    # change.
-    dest_path = osp.join(data_dir, 'README.txt')
-    if not os.path.isfile(dest_path):
-        shutil.copy2(osp.join(skimage_distribution_dir, 'data', 'README.txt'),
-                     dest_path)
-
-    # Fetch all legacy data so that it is available by default
-    for filename in legacy_registry:
-        _fetch(filename)
-
-
-# This function creates directories, and has been the source of issues for
-# downstream users, see
-# https://github.com/scikit-image/scikit-image/issues/4660
-# https://github.com/scikit-image/scikit-image/issues/4664
-if has_pooch:
-    _init_pooch()
 
 
 def download_all(directory=None):
@@ -307,10 +282,18 @@ def download_all(directory=None):
     old_dir = image_fetcher.path
     try:
         if directory is not None:
+            directory = osp.expanduser(directory)
             image_fetcher.path = directory
+        _ensure_cache_dir(target_dir=image_fetcher.path)
+
+        for data_filename in image_fetcher.registry:
+            file_path = _fetch(data_filename)
 
-        for filename in image_fetcher.registry:
-            _fetch(filename)
+            # Copy to `directory` or implicit cache if it is not already there
+            if not file_path.startswith(str(image_fetcher.path)):
+                dest_path = osp.join(image_fetcher.path, data_filename)
+                os.makedirs(osp.dirname(dest_path), exist_ok=True)
+                shutil.copy2(file_path, dest_path)
     finally:
         image_fetcher.path = old_dir
 


Test patch:
diff --git a/skimage/data/tests/test_data.py b/skimage/data/tests/test_data.py
--- a/skimage/data/tests/test_data.py
+++ b/skimage/data/tests/test_data.py
@@ -7,13 +7,6 @@
 import pytest
 
 
-def test_data_dir():
-    # data_dir should be a directory people can use as a standard directory
-    # https://github.com/scikit-image/scikit-image/pull/3945#issuecomment-498141893
-    data_dir = data.data_dir
-    assert 'astronaut.png' in os.listdir(data_dir)
-
-
 def test_download_all_with_pooch():
     # jni first wrote this test with the intention of
     # fully deleting the files in the data_dir,
@@ -30,6 +23,7 @@ def test_download_all_with_pooch():
     data_dir = data.data_dir
     if image_fetcher is not None:
         data.download_all()
+        assert 'astronaut.png' in os.listdir(data_dir)
         assert len(os.listdir(data_dir)) > 50
     else:
         with pytest.raises(ModuleNotFoundError):
diff --git a/skimage/io/tests/test_multi_image.py b/skimage/io/tests/test_multi_image.py
--- a/skimage/io/tests/test_multi_image.py
+++ b/skimage/io/tests/test_multi_image.py
@@ -24,6 +24,7 @@ def imgs():
 
     reset_plugins()
 
+
 def test_shapes(imgs):
     imgs = imgs[-1]
     assert imgs[0][0].shape == imgs[0][1].shape