Repository: joblib/joblib
Pull request #1366 (2022-11-24 18:44:31)

Fix memmapping_reducer when 'os' has no attribute 'statvfs'
Closes https://github.com/joblib/joblib/issues/902

In this case when 'os' has no attribute 'statvfs' we don't use `/dev/shm` even if it exists, as we are then not able to determine how much free space it has.

The other possible solution could be to ignore `SYSTEM_SHARED_MEM_FS_MIN_SIZE` and still write to `/dev/shm`.

The typical use case where this could happen is when running a Linux like system on Windows (which doesn't have statvfs apparently). I'm not sure what would be better.



codecov[bot]: # [Codecov](https://codecov.io/gh/joblib/joblib/pull/1366?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib) Report
Base: **93.92**% // Head: **70.79**% // Decreases project coverage by **`-23.12%`** :warning:
> Coverage data is based on head [(`6b84931`)](https://codecov.io/gh/joblib/joblib/pull/1366?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib) compared to base [(`520dca5`)](https://codecov.io/gh/joblib/joblib/commit/520dca5a9f53b817f046d2beabd75172c9618a5e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib).
> Patch coverage: 88.88% of modified lines in pull request are covered.

> :exclamation: Current head 6b84931 differs from pull request most recent head b294f3a. Consider uploading reports for the commit b294f3a to get more accurate results

<details><summary>Additional details and impacted files</summary>


```diff
@@             Coverage Diff             @@
##           master    #1366       +/-   ##
===========================================
- Coverage   93.92%   70.79%   -23.13%     
===========================================
  Files          52       52               
  Lines        7291     7112      -179     
===========================================
- Hits         6848     5035     -1813     
- Misses        443     2077     +1634     
```


| [Impacted Files](https://codecov.io/gh/joblib/joblib/pull/1366?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib) | Coverage Δ | |
|---|---|---|
| [joblib/test/test\_memmapping.py](https://codecov.io/gh/joblib/joblib/pull/1366/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib#diff-am9ibGliL3Rlc3QvdGVzdF9tZW1tYXBwaW5nLnB5) | `35.17% <87.50%> (-64.06%)` | :arrow_down: |
| [joblib/\_memmapping\_reducer.py](https://codecov.io/gh/joblib/joblib/pull/1366/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib#diff-am9ibGliL19tZW1tYXBwaW5nX3JlZHVjZXIucHk=) | `38.72% <100.00%> (-57.16%)` | :arrow_down: |
| [joblib/test/test\_dask.py](https://codecov.io/gh/joblib/joblib/pull/1366/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib#diff-am9ibGliL3Rlc3QvdGVzdF9kYXNrLnB5) | `3.92% <0.00%> (-87.27%)` | :arrow_down: |
| [joblib/executor.py](https://codecov.io/gh/joblib/joblib/pull/1366/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib#diff-am9ibGliL2V4ZWN1dG9yLnB5) | `30.95% <0.00%> (-69.05%)` | :arrow_down: |
| [joblib/\_dask.py](https://codecov.io/gh/joblib/joblib/pull/1366/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib#diff-am9ibGliL19kYXNrLnB5) | `24.01% <0.00%> (-68.67%)` | :arrow_down: |
| [joblib/\_multiprocessing\_helpers.py](https://codecov.io/gh/joblib/joblib/pull/1366/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib#diff-am9ibGliL19tdWx0aXByb2Nlc3NpbmdfaGVscGVycy5weQ==) | `26.92% <0.00%> (-61.54%)` | :arrow_down: |
| [joblib/pool.py](https://codecov.io/gh/joblib/joblib/pull/1366/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib#diff-am9ibGliL3Bvb2wucHk=) | `27.04% <0.00%> (-60.76%)` | :arrow_down: |
| [joblib/test/test\_module.py](https://codecov.io/gh/joblib/joblib/pull/1366/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib#diff-am9ibGliL3Rlc3QvdGVzdF9tb2R1bGUucHk=) | `52.94% <0.00%> (-47.06%)` | :arrow_down: |
| ... and [29 more](https://codecov.io/gh/joblib/joblib/pull/1366/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib) | |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib)

</details>

[:umbrella: View full report at Codecov](https://codecov.io/gh/joblib/joblib/pull/1366?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib).


Patch:
diff --git a/CHANGES.rst b/CHANGES.rst
index 63e9559..72e983e 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -11,6 +11,10 @@ In development
   JOBLIB_TESTS_DEFAULT_PARALLEL_BACKEND environment variable.
   https://github.com/joblib/joblib/pull/1356
 
+- Fix temporary folder creation in `joblib.Parallel` on Linux subsystems on Windows
+  which do have `/dev/shm` but don't have the `os.statvfs` function 
+  https://github.com/joblib/joblib/issues/1353
+
 - Drop runtime dependency on ``distutils``. ``distutils`` is going away
   in Python 3.12 and is deprecated from Python 3.10 onwards. This import
   was kept around to avoid breaking scikit-learn, however it's now been
diff --git a/joblib/_memmapping_reducer.py b/joblib/_memmapping_reducer.py
index 9d350c0..88b1091 100644
--- a/joblib/_memmapping_reducer.py
+++ b/joblib/_memmapping_reducer.py
@@ -203,7 +203,7 @@ def _get_temp_dir(pool_folder_name, temp_folder=None):
     if temp_folder is None:
         temp_folder = os.environ.get('JOBLIB_TEMP_FOLDER', None)
     if temp_folder is None:
-        if os.path.exists(SYSTEM_SHARED_MEM_FS):
+        if os.path.exists(SYSTEM_SHARED_MEM_FS) and hasattr(os, 'statvfs'):
             try:
                 shm_stats = os.statvfs(SYSTEM_SHARED_MEM_FS)
                 available_nbytes = shm_stats.f_bsize * shm_stats.f_bavail


Test patch:
diff --git a/joblib/test/test_memmapping.py b/joblib/test/test_memmapping.py
index bdc825f..4d298d0 100644
--- a/joblib/test/test_memmapping.py
+++ b/joblib/test/test_memmapping.py
@@ -1058,6 +1058,23 @@ def test_pool_get_temp_dir(tmpdir):
     assert pool_folder.endswith(pool_folder_name)
 
 
+def test_pool_get_temp_dir_no_statvfs(tmpdir, monkeypatch):
+    """Check that _get_temp_dir works when os.statvfs is not defined
+
+    Regression test for #902
+    """
+    pool_folder_name = 'test.tmpdir'
+    import joblib._memmapping_reducer
+    if hasattr(joblib._memmapping_reducer.os, 'statvfs'):
+        # We are on Unix, since Windows doesn't have this function
+        monkeypatch.delattr(joblib._memmapping_reducer.os, 'statvfs')
+
+    pool_folder, shared_mem = _get_temp_dir(pool_folder_name, temp_folder=None)
+    if sys.platform.startswith('win'):
+        assert shared_mem is False
+    assert pool_folder.endswith(pool_folder_name)
+
+
 @with_numpy
 @skipif(sys.platform == 'win32', reason='This test fails with a '
         'PermissionError on Windows')