Repository: DataBiosphere/toil

Issue:
Toil worker deadlocks looking for orphaned deferred functions in single machine mode when the workflow directory is on NFS
I'm running toil-vg on Courtyard, with TMP_DIR (and thus the workflow directory) on our group NFS mount, because Courtyard doesn't (or didn't) think it has enough temp disk space. I am also running with caching enabled.

I'm noticing that my workflow hangs until I restart it, at which point it makes a little more progress and hangs again. When I ctrl+c the workflow, and run `toil status --printPerJobStats ./trash/tree`, and look at some of the log files it says jobs have, they have things like this in them:

```
Traceback (most recent call last):
  File "/public/groups/cgl/graph-genomes/anovak/build/toil-vg/venv2/lib/python2.7/site-packages/toil/worker.py", line 369, in workerScript
    fileStore.startCommit(jobState=False)
  File "/public/home/anovak/.local/lib64/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/public/groups/cgl/graph-genomes/anovak/build/toil-vg/venv2/lib/python2.7/site-packages/toil/deferred.py", line 188, in open
    self._runOrphanedDeferredFunctions()
  File "/public/groups/cgl/graph-genomes/anovak/build/toil-vg/venv2/lib/python2.7/site-packages/toil/deferred.py", line 310, in _runOrphanedDeferredFunctions
    fcntl.lockf(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
KeyboardInterrupt
```

Or:

```
Traceback (most recent call last):
  File "/public/groups/cgl/graph-genomes/anovak/build/toil-vg/venv2/lib/python2.7/site-packages/toil/worker.py", line 369, in workerScript
    fileStore.startCommit(jobState=False)
  File "/public/home/anovak/.local/lib64/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/public/groups/cgl/graph-genomes/anovak/build/toil-vg/venv2/lib/python2.7/site-packages/toil/deferred.py", line 188, in open
    self._runOrphanedDeferredFunctions()
  File "/public/groups/cgl/graph-genomes/anovak/build/toil-vg/venv2/lib/python2.7/site-packages/toil/deferred.py", line 304, in _runOrphanedDeferredFunctions
    fd = os.open(fullFilename, os.O_RDWR)
KeyboardInterrupt
```

I think the problem is that the deferred function code expects its deletes to take, which is how it marks deferred functions as done. But it deletes files while they are open (and locked, so that nobody else can race the delete), which on NFS drops them back in the directory under a new name. Either trying to open/lock the `.nfs*` files is hanging, or I have multiple Toil workers constantly opening/reopening the files so they don't get cleaned up, or `close()`-ing a Python File object that was `fdopen()`'d from a file descriptor isn't actually closing the backing file descriptor, and the worker is fighting itself over files it itself still has open.

The easy solution is to ignore `.nfs*` files in the deferred functions directory.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-494)
┆Issue Number: TOIL-494



Discussion:

Resolving patch:
diff --git a/src/toil/deferred.py b/src/toil/deferred.py
--- a/src/toil/deferred.py
+++ b/src/toil/deferred.py
@@ -99,6 +99,15 @@ class DeferredFunctionManager(object):
     be locked, and will take them over.
     """
 
+    # Define what directory the state directory should actaully be, under the base
+    STATE_DIR_STEM = 'deferred'
+    # Have a prefix to distinguish our deferred functions from e.g. NFS
+    # "silly rename" files, or other garbage that people put in our
+    # directory
+    PREFIX = 'func'
+    # And a suffix to distingusidh in-progress from completed files
+    WIP_SUFFIX = '.tmp'
+
     def __init__(self, stateDirBase):
         """
         Create a new DeferredFunctionManager, sharing state with other
@@ -115,12 +124,14 @@ def __init__(self, stateDirBase):
         """
 
         # Work out where state files live
-        self.stateDir = os.path.join(stateDirBase, "deferred")
+        self.stateDir = os.path.join(stateDirBase, self.STATE_DIR_STEM)
         mkdir_p(self.stateDir)
 
         # We need to get a state file, locked by us and not somebody scanning for abandoned state files.
-        # So we suffix not-yet-ready ones with .tmp
-        self.stateFD, self.stateFileName = tempfile.mkstemp(dir=self.stateDir, suffix='.tmp')
+        # So we suffix not-yet-ready ones with our suffix
+        self.stateFD, self.stateFileName = tempfile.mkstemp(dir=self.stateDir,
+                                                            prefix=self.PREFIX,
+                                                            suffix=self.WIP_SUFFIX)
 
         # Lock the state file. The lock will automatically go away if our process does.
         try:
@@ -129,9 +140,9 @@ def __init__(self, stateDirBase):
             # Someone else might have locked it even though they should not have.
             raise RuntimeError("Could not lock deferred function state file %s: %s" % (self.stateFileName, str(e)))
 
-        # Rename it to remove the ".tmp"
-        os.rename(self.stateFileName, self.stateFileName[:-4])
-        self.stateFileName = self.stateFileName[:-4]
+        # Rename it to remove the suffix
+        os.rename(self.stateFileName, self.stateFileName[:-len(self.WIP_SUFFIX)])
+        self.stateFileName = self.stateFileName[:-len(self.WIP_SUFFIX)]
 
         # Wrap the FD in a Python file object, which we will use to actually use it.
         # Problem: we can't be readable and writable at the same time. So we need two file objects.
@@ -208,7 +219,7 @@ def cleanupWorker(cls, stateDirBase):
         # Clean up the directory we have been using.
         # It might not be empty if .tmp files escaped: nobody can tell they
         # aren't just waiting to be locked.
-        shutil.rmtree(os.path.join(stateDirBase, "deferred"))
+        shutil.rmtree(os.path.join(stateDirBase, cls.STATE_DIR_STEM))
 
 
 
@@ -280,10 +291,14 @@ def _runOrphanedDeferredFunctions(self):
             for filename in os.listdir(self.stateDir):
                 # Scan the whole directory for work nobody else owns.
 
-                if filename.endswith(".tmp"):
+                if filename.endswith(self.WIP_SUFFIX):
                     # Skip files from instances that are still being set up
                     continue
 
+                if not filename.startswith(self.PREFIX):
+                    # Skip NFS deleted files and any other contaminants
+                    continue
+
                 fullFilename = os.path.join(self.stateDir, filename)
 
                 if fullFilename == self.stateFileName:
@@ -332,7 +347,8 @@ def _runOrphanedDeferredFunctions(self):
                 # Unlock it
                 fcntl.lockf(fd, fcntl.LOCK_UN)
 
-                # Now close it.
+                # Now close it. This closes the backing file descriptor. See
+                # <https://stackoverflow.com/a/24984929>
                 fileObj.close()
 
                 


Test patch: