Repository: Project-MONAI/MONAI

Issue:
`test_persistentdataset_dist` error on windows
**Describe the bug**
the error occurs randomly:
```
test_mp_dataset (tests.test_persistentdataset_dist.TestDistDataset) ... Process SpawnProcess-51:
persistent 1
Traceback (most recent call last):
  File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\shutil.py", line 791, in move
    os.rename(src, real_dst)
FileExistsError: [WinError 183] Cannot create a file when that file already exists: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmp2ndb3qxg\\54c0443faf33e022375c4a3adc6afd11.pt' -> 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpovt2z4gf\\test\\54c0443faf33e022375c4a3adc6afd11.pt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\multiprocessing\process.py", line 315, in _bootstrap
    self.run()
  File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "D:\a\MONAI\MONAI\tests\utils.py", line 396, in run_process
    raise e
  File "D:\a\MONAI\MONAI\tests\utils.py", line 387, in run_process
    func(*args, **kwargs)
  File "D:\a\MONAI\MONAI\tests\utils.py", line 557, in _call_original_func
    return f(*args, **kwargs)
  File "D:\a\MONAI\MONAI\tests\test_persistentdataset_dist.py", line 73, in test_mp_dataset
    self.assertEqual(list(ds1), list(ds))
  File "D:\a\MONAI\MONAI\monai\data\dataset.py", line 97, in __getitem__
    return self._transform(index)
  File "D:\a\MONAI\MONAI\monai\data\dataset.py", line 360, in _transform
    pre_random_item = self._cachecheck(self.data[index])
  File "D:\a\MONAI\MONAI\monai\data\dataset.py", line 354, in _cachecheck
    shutil.move(temp_hash_file, hashfile)
  File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\shutil.py", line 811, in move
    copy_function(src, real_dst)
  File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\shutil.py", line 435, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\shutil.py", line 264, in copyfile
    with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
PermissionError: [Errno 13] Permission denied: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpovt2z4gf\\test\\54c0443faf33e022375c4a3adc6afd11.pt'
persistent 0
FAIL
```

--- update --- 
A similar issue was reported at [bpo-29982](https://bugs.python.org/issue29982) where the calling app `tempfile.TemporaryDirectory` to `shutil` should handle the exception. A python PR has been merged to address that issue https://github.com/python/cpython/pull/24793.
Before that PR is officially released in python 3.10, we handle the exception with another `try...except` in https://github.com/Project-MONAI/MONAI/pull/3633.


Discussion:
Hi @wyli ,

I remember I saw this issue before, let me try to find it out.

Thanks.

thanks, I put a patch here: https://github.com/Project-MONAI/MONAI/pull/3615
Hi @wyli ,

May I know where you found this error? It always happens or only rarely happens in CI env?

Thanks.

the most recent error is here https://github.com/Project-MONAI/MONAI/runs/4748467145?check_suite_focus=true. I think it becomes more frequent after this testing script update: https://github.com/Project-MONAI/MONAI/commit/7f23f38e7e7404eb86759784789a9dd03478a96c#diff-8b8238839d9c98a26d9a1d5a504a38807afa0dbee153e4d565de8e617e98e51f
OK, do you have any idea what's update in this `diff` made it more frequent? I didn't see any obvious thing.

Thanks.

Hi @wyli ,

I checked some pages about this error, maybe it's related to: The test program doesn't have permission to create file on windows `C:` section: https://www.cnblogs.com/mstk/p/14724509.html
Do you know how to set our test program of windows CI to run as admin?

Thanks.

Hi @wyli ,

I just saw this issue again when running CI for PR https://github.com/Project-MONAI/MONAI/pull/3625.
Do you think it's related to my above analysis about windows admin permission when the `temp` is randomly on C: section?

Thanks.

looks like a known issue https://github.com/python/cpython/pull/24793 and python 3.10 has an option `ignore_cleanup_errors` to address it.
> looks like a known issue [python/cpython#24793](https://github.com/python/cpython/pull/24793) and python 3.10 has an option `ignore_cleanup_errors` to address it.

Hi @wyli ,

Thanks for your investigation, so if someone faces this issue, we can ask him to update python?


Resolving patch:
diff --git a/monai/data/dataset.py b/monai/data/dataset.py
--- a/monai/data/dataset.py
+++ b/monai/data/dataset.py
@@ -335,7 +335,9 @@ def _cachecheck(self, item_transformed):
                     raise e
 
         _item_transformed = self._pre_transform(deepcopy(item_transformed))  # keep the original hashed
-        if hashfile is not None:
+        if hashfile is None:
+            return _item_transformed
+        try:
             # NOTE: Writing to a temporary directory and then using a nearly atomic rename operation
             #       to make the cache more robust to manual killing of parent process
             #       which may leave partially written cache files in an incomplete state
@@ -354,6 +356,8 @@ def _cachecheck(self, item_transformed):
                         shutil.move(temp_hash_file, hashfile)
                     except FileExistsError:
                         pass
+        except PermissionError:  # project-monai/monai issue #3613
+            pass
         return _item_transformed
 
     def _transform(self, index: int):


Test patch: