Repository: fatiando/pooch

Issue:
Crash on import when running in parallel
**Description of the problem**
<!--
Please be as detailed as you can when describing an issue. The more information
we have, the easier it will be for us to track this down.
-->

We recommend a workflow of keeping a `Pooch` as a global variable by calling `pooch.create` at import time. This is problematic when running in parallel and the cache folder has not been created yet. Each parallel process/thread will try to create it at the same time leading to a crash.

For our users, this is particularly bad since their packages crash on `import` even if there is no use of the sample datasets. 

Not sure exactly what we can do on our end except:

1. Add a warning in the docs about this issue.
2. Change our recommended workflow to wrapping `pooch.create` in a function. This way the crash wouldn't happen at `import` but only when trying to fetch a dataset in parallel. This gives packages a chance to warn users not to load sample data for the first time in parallel.

Perhaps an ideal solution would be for Pooch to create the cache folder at install time. But that would mean hooking into `setup.py` (which is complicated) and I don't know if we could even do this with conda.

**Full code that generated the error**

See scikit-image/scikit-image#4660


Comments:

Patch:
diff --git a/pooch/utils.py b/pooch/utils.py
index 841ad0c..fa88114 100644
--- a/pooch/utils.py
+++ b/pooch/utils.py
@@ -253,12 +253,21 @@ def make_local_storage(path, env=None, version=None):
     try:
         if not os.path.exists(path):
             action = "create"
-            os.makedirs(path)
+            # When running in parallel, it's possible that multiple jobs will
+            # try to create the path at the same time. Use exist_ok to avoid
+            # raising an error.
+            os.makedirs(path, exist_ok=True)
         else:
             action = "write to"
             with tempfile.NamedTemporaryFile(dir=path):
                 pass
     except PermissionError:
+        # Only log an error message instead of raising an exception. The cache
+        # is usually created at import time, so raising an exception here would
+        # cause packages to crash immediately, even if users aren't using the
+        # sample data at all. So issue a warning here just in case and only
+        # crash with an exception when the user actually tries to download
+        # data (Pooch.fetch or retrieve).
         message = (
             "Cannot %s data cache folder '%s'. "
             "Will not be able to download remote data files. "


Test patch:
diff --git a/pooch/tests/test_utils.py b/pooch/tests/test_utils.py
index 4e6501f..536db26 100644
--- a/pooch/tests/test_utils.py
+++ b/pooch/tests/test_utils.py
@@ -2,10 +2,13 @@
 Test the utility functions.
 """
 import os
+import shutil
 import hashlib
+import time
 from pathlib import Path
 import tempfile
 from tempfile import NamedTemporaryFile, TemporaryDirectory
+from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
 
 import pytest
 
@@ -41,6 +44,45 @@ def test_unique_name_long():
     assert fname.split("-")[1][:10] == "aaaaaaaaaa"
 
 
+@pytest.mark.parametrize(
+    "pool", [ThreadPoolExecutor, ProcessPoolExecutor], ids=["threads", "processes"],
+)
+def test_make_local_storage_parallel(pool, monkeypatch):
+    "Try to create the cache folder in parallel"
+    # Can cause multiple attempts at creating the folder which leads to an
+    # exception. Check that this doesn't happen.
+    # See https://github.com/fatiando/pooch/issues/170
+
+    # Monkey path makedirs to make it delay before creating the directory.
+    # Otherwise, the dispatch is too fast and the directory will exist before
+    # another process tries to create it.
+
+    # Need to keep a reference to the original function to avoid infinite
+    # recursions from the monkey patching.
+    makedirs = os.makedirs
+
+    def mockmakedirs(path, exist_ok=False):  # pylint: disable=unused-argument
+        "Delay before calling makedirs"
+        time.sleep(1.5)
+        makedirs(path, exist_ok=exist_ok)
+
+    monkeypatch.setattr(os, "makedirs", mockmakedirs)
+
+    data_cache = os.path.join(os.curdir, "test_parallel_cache")
+    assert not os.path.exists(data_cache)
+
+    try:
+        with pool() as executor:
+            futures = [
+                executor.submit(make_local_storage, data_cache) for i in range(4)
+            ]
+            for future in futures:
+                assert os.path.exists(str(future.result()))
+    finally:
+        if os.path.exists(data_cache):
+            shutil.rmtree(data_cache)
+
+
 def test_local_storage_makedirs_permissionerror(monkeypatch):
     "Should warn the user when can't create the local data dir"