Repository: DataBiosphere/toil

Issue:
Clean up errors in a NFS file system
During workflow execution (CWL), toil-cwl-runner sometimes fails to clean up used directories as it is targeting the ".nfsXXXXX" files:

```
[2021-08-23T22:16:44+0100] [MainThread] [D] [toil.lib.threading] PID 2618437 releasing mutex /hps/nobackup/assemblies/ERP1193/ERP119312/toil/work-dir/toil-mutex-3c8fe4b5-39af-4ccb-b639-5e115aa81cf9-cleanup-arena-lock
Traceback (most recent call last):
  File "/hps/software/miniconda_py39/envs/toil-5.4.0/bin/_toil_worker", line 8, in <module>
    sys.exit(main())
  File "/hps/software/miniconda_py39/envs/toil-5.4.0/lib/python3.9/site-packages/toil/worker.py", line 685, in main
    exit_code = workerScript(jobStore, config, options.jobName, options.jobStoreID)
  File "/hps/software/miniconda_py39/envs/toil-5.4.0/lib/python3.9/contextlib.py", line 124, in __exit__
    next(self.gen)
  File "/hps/software/miniconda_py39/envs/toil-5.4.0/lib/python3.9/site-packages/toil/worker.py", line 661, in in_contexts
    yield
  File "/hps/software/miniconda_py39/envs/toil-5.4.0/lib/python3.9/site-packages/toil/batchSystems/abstractBatchSystem.py", line 501, in __exit__
    BatchSystemSupport.workerCleanup(self.workerCleanupInfo)
  File "/hps/software/miniconda_py39/envs/toil-5.4.0/lib/python3.9/site-packages/toil/batchSystems/abstractBatchSystem.py", line 380, in workerCleanup
    DeferredFunctionManager.cleanupWorker(workflowDir)
  File "/hps/software/miniconda_py39/envs/toil-5.4.0/lib/python3.9/site-packages/toil/deferred.py", line 216, in cleanupWorker
    shutil.rmtree(os.path.join(stateDirBase, cls.STATE_DIR_STEM))
  File "/hps/software/miniconda_py39/envs/toil-5.4.0/lib/python3.9/shutil.py", line 719, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/hps/software/miniconda_py39/envs/toil-5.4.0/lib/python3.9/shutil.py", line 676, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "/hps/software/miniconda_py39/envs/toil-5.4.0/lib/python3.9/shutil.py", line 674, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '.nfs4f0e00ff85f63eb50002fa9c'
```

I am using toil-5.4.0, please let me know if you need more details. Thanks

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-998)
┆Issue Number: TOIL-998



Discussion:
It looks like Toil is trying to delete the local directory used for our "deferred functions" here, and failing because a file in it is open. Normal filesystems let you delete open files, but on NFS an attempt to delete an open file will turn it into a `.nfsXXXXXXX` file which cannot be deleted, so it is impossible to delete a directory tree that contains any open files.

It might be that we don't *actually* manage to close all the files we had open to look for deferred functions in when we try to do it here:
https://github.com/DataBiosphere/toil/blob/81ca29c797bb802e57088b02717cf6695d9bdc2f/src/toil/deferred.py#L210-L211
Maybe we need some code to actually go in and close all the file handles that is missing?

This is all happening in a directory that Toil means to be local to the executing node, though; in addition to deferred functions we use this local workflow directory tree to do things like hold local caches of files, so it's not advised to have it on NFS. `toil-cwl-runner` is set up to put everything in one hierarchy by default, I believe somewhere in the temp directory, but if that is shared storage you will want to use the `--workDir` option to pass the path to non-shared temporary storage available on each node. Using that option to pass a non-NFS place to put these files should work.
It might be this logic that doesn't play well with NFS:

https://github.com/DataBiosphere/toil/blob/81ca29c797bb802e57088b02717cf6695d9bdc2f/src/toil/deferred.py#L147-L163

We always delete the file, then unlock it through an open file descriptor, and then wait for a Python file object we made around the file descriptor to get around to closing it when it is GC'd as a member of the object here. This is always going to make `.nfs` files, and I'm not sure whether they can be relied upon to go away promptly when the file descriptors to them actually close, or whether we really can rely on the file descriptors to be closed when we `del` the containing object.

We might need to revise this process to use a non-`__del__` method that renames the files (instead of deleting them), then closes them manually, and then deletes them.
I was checking where can I set workDir, our cluster nodes have /tmp with ~200GB that can be enough for some runs, however, large runs will be complicated.
I tested running toil with options `--cleanWorkDir never --clean never` and still got those cleaning errors.
 
Hello,

We also had this type of problem with the Jobstrore using a GPFS/NFS shared partition with Toil 5.3.0, when using option --runCwlInternalJobsOnWorkers.

Can it be the same issue ?

```
[2021-07-07T06:10:20+0000] [MainThread] [D] [toil.common] ... finished shutting down the batch system in 0.01743769645690918 seconds.
Traceback (most recent call last):
  File "/usr/local/bin/toil-cwl-runner", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/toil/cwl/cwltoil.py", line 2383, in main
    outobj = toil.start(wf1)
  File "/usr/local/lib/python3.9/site-packages/toil/common.py", line 823, in start
    return self._runMainLoop(rootJobDescription)
  File "/usr/local/lib/python3.9/site-packages/toil/common.py", line 1111, in _runMainLoop
    return Leader(config=self.config,
  File "/usr/local/lib/python3.9/site-packages/toil/leader.py", line 228, in run
    self.innerLoop()
  File "/usr/local/lib/python3.9/site-packages/toil/leader.py", line 615, in innerLoop
    self._gatherUpdatedJobs(updatedJobTuple)
  File "/usr/local/lib/python3.9/site-packages/toil/leader.py", line 574, in _gatherUpdatedJobs
    self.processFinishedJob(jobID, exitStatus, wallTime=wallTime, exitReason=exitReason)
  File "/usr/local/lib/python3.9/site-packages/toil/leader.py", line 960, in processFinishedJob
    replacementJob = self.jobStore.load(jobStoreID)
  File "/usr/local/lib/python3.9/site-packages/toil/jobStores/fileJobStore.py", line 210, in load
    with open(jobFile, 'rb') as fileHandle:
FileNotFoundError: [Errno 2] No such file or directory: '/GPFS_PATH/job_stores/20210707T060908.756757/jobs/kind-ResolveIndirect/instance-7d5q23l4/job'
```

Thanks
> I was checking where can I set workDir, our cluster nodes have /tmp with ~200GB that can be enough for some runs, however, large runs will be complicated.
> I tested running toil with options `--cleanWorkDir never --clean never` and still got those cleaning errors.

Another issue with using a "local" workdir is that logs are not copied over so it's very hard to debug if something goes wrong.
@Guigzai Your issue is different; I think it's a race between your filesystem and your scheduler, where the scheduler says that a job is done before all its on-disk writes are visible to the leader through the filesystem. It looks a lot like #1153 which we closed as stale, but can actually still happen.

I'll repopen that issue for you.

Resolving patch:
diff --git a/src/toil/deferred.py b/src/toil/deferred.py
--- a/src/toil/deferred.py
+++ b/src/toil/deferred.py
@@ -21,6 +21,7 @@
 
 import dill
 
+from toil.lib.io import robust_rmtree
 from toil.realtimeLogger import RealtimeLogger
 from toil.resource import ModuleDescriptor
 
@@ -210,11 +211,11 @@ def cleanupWorker(cls, stateDirBase):
         # Close all the files in there.
         del cleaner
 
-        # Clean up the directory we have been using.
-        # It might not be empty if .tmp files escaped: nobody can tell they
-        # aren't just waiting to be locked.
-        shutil.rmtree(os.path.join(stateDirBase, cls.STATE_DIR_STEM))
-
+        try:
+            robust_rmtree(os.path.join(stateDirBase, cls.STATE_DIR_STEM))
+        except OSError as err:
+            logger.exception(err)
+            # we tried, lets move on
 
 
     def _runDeferredFunction(self, deferredFunction):


Test patch: