Repository: DataBiosphere/toil

Issue:
Directory not empty error when trying to delete the job store results in Toil raising an exception to its caller
I have some code in toil-vg that calls into Toil to run a workflow. Toil used to work fine, but today it's decided to throw a directory not empty error when trying to delete its job store. I suspect it's something to do with our `hive` mount not quite giving the read-your-own-writes consistency guarantees we are supposed to have in a filesystem; when I go to the directory to see the offending file, it's empty andI can delete it just fine. Alternately, some other thread or even a completely different process could be dumping extra files into Toil's temp directories as it is cleaning up.

Either way, an unexpected file in the temp directories shouldn't make the Toil run throw (and kill my calling code). Toil should clean up the single machine mode job store as best it can, and if someone is dropping files in it as Toil is deleting it, Toil should delete those new files instead of crashing out when it wanted the directory to be empty and it isn't.

Here's the error I'm getting through toil-vg:

```
args = (<vgci.VGCITest testMethod=test_sim_mhc_snp1kg>,), kwargs = {}
new_seconds = 3600, old = 0
    @wraps(function)
    def new_function(*args, **kwargs):
        new_seconds = kwargs.pop('timeout', seconds)
        if new_seconds:
            old = signal.signal(signal.SIGALRM, handler)
            signal.setitimer(signal.ITIMER_REAL, new_seconds)
        try:
>           return function(*args, **kwargs)
.env/lib/python2.7/site-packages/timeout_decorator/timeout_decorator.py:81:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
jenkins/vgci.py:968: in test_sim_mhc_snp1kg
    source_path_names=['GI568335986', 'GI568335994'], acc_threshold=0.02,
auc_threshold=0.02)
jenkins/vgci.py:941: in _test_mapeval
    test_graphs, score_baseline_graph, multipath, paired_only, sim_opts,
sim_fastq, tag)
jenkins/vgci.py:548: in _mapeval_vg_run
    returned = toil.start(main_job)
.env/lib/python2.7/site-packages/toil/common.py:666: in __exit__
    self._jobStore.destroy()
.env/lib/python2.7/site-packages/toil/jobStores/fileJobStore.py:83: in destroy
    shutil.rmtree(self.jobStoreDir)
../../python/lib/python2.7/shutil.py:247: in rmtree
    rmtree(fullname, ignore_errors, onerror)
../../python/lib/python2.7/shutil.py:247: in rmtree
    rmtree(fullname, ignore_errors, onerror)
../../python/lib/python2.7/shutil.py:247: in rmtree
    rmtree(fullname, ignore_errors, onerror)
../../python/lib/python2.7/shutil.py:256: in rmtree
    onerror(os.rmdir, path, sys.exc_info())
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
path = '/cluster/home/anovak/build/vg/vgci-work/jobstoresim-MHC-cactus/tmp/i/K'
ignore_errors = False, onerror = <function onerror at 0x7f399d93db18>
    def rmtree(path, ignore_errors=False, onerror=None):
        """Recursively delete a directory tree.
        If ignore_errors is set, errors are ignored; otherwise, if onerror
        is set, it is called to handle the error with arguments (func,
        path, exc_info) where func is os.listdir, os.remove, or os.rmdir;
        path is the argument to that function that caused it to fail; and
        exc_info is a tuple returned by sys.exc_info().  If ignore_errors
        is false and onerror is None, an exception is raised.
        """
        if ignore_errors:
            def onerror(*args):
                pass
        elif onerror is None:
            def onerror(*args):
                raise
        try:
            if os.path.islink(path):
                # symlinks to directories are forbidden, see bug #1669
                raise OSError("Cannot call rmtree on a symbolic link")
        except OSError:
            onerror(os.path.islink, path, sys.exc_info())
            # can't continue even if onerror hook returns
            return
        names = []
        try:
            names = os.listdir(path)
        except os.error, err:
            onerror(os.listdir, path, sys.exc_info())
        for name in names:
            fullname = os.path.join(path, name)
            try:
                mode = os.lstat(fullname).st_mode
            except os.error:
                mode = 0
            if stat.S_ISDIR(mode):
                rmtree(fullname, ignore_errors, onerror)
            else:
                try:
                    os.remove(fullname)
                except os.error, err:
                    onerror(os.remove, fullname, sys.exc_info())
        try:
>           os.rmdir(path)
E           OSError: [Errno 39] Directory not empty:
'/cluster/home/anovak/build/vg/vgci-work/jobstoresim-MHC-cactus/tmp/i/K'
../../python/lib/python2.7/shutil.py:254: OSError

Failure Message

OSError: [Errno 39] Directory not empty: '/cluster/home/anovak/build/vg/vgci-
work/jobstoresim-MHC-cactus/tmp/i/K'
```


Discussion:
I just encountered this error again today; it affects every local run of the vg performance tests now for some reason.

Job store destruction for the fileJobStore needs to be made robust to not-quite-consistent filesystems, either by mashing rmtree until it succeeds, calling out to `rm`, or just warning about the failed cleanup and pretending the workflow succeeded.
The underlying problem is basically the same as https://github.com/hashdist/hashdist/issues/113 AFAIK.

Resolving patch:
diff --git a/src/toil/jobStores/fileJobStore.py b/src/toil/jobStores/fileJobStore.py
--- a/src/toil/jobStores/fileJobStore.py
+++ b/src/toil/jobStores/fileJobStore.py
@@ -27,6 +27,7 @@
 import tempfile
 import stat
 import errno
+import time
 import traceback
 try:
     import cPickle as pickle
@@ -34,6 +35,7 @@
     import pickle
 
 # toil and bd2k dependencies
+from bd2k.util.files import rm_f
 from toil.fileStore import FileID
 from toil.lib.bioio import absSymPath
 from toil.jobStores.abstractJobStore import (AbstractJobStore,
@@ -70,6 +72,9 @@ def __init__(self, path):
         # Directory where temporary files go
         self.tempFilesDir = os.path.join(self.jobStoreDir, 'tmp')
         self.linkImports = None
+        
+        # This flag is used by self._recursiveDelete to limit warning count
+        self._warnedOnNfs = False
 
     def initialize(self, config):
         try:
@@ -87,10 +92,76 @@ def resume(self):
         if not os.path.isdir(self.jobStoreDir):
             raise NoSuchJobStoreException(self.jobStoreDir)
         super(FileJobStore, self).resume()
+        
+    def _recursiveDelete(self, path, maxTries=3):
+        """
+        
+        Delete a directory recursively, retrying after a delay on error, to
+        deal with filesystems that present a bad FS abstraction.
+        
+        Detect and warn about .nfs files.
+        
+        If the directory cannot ultiumately be removed after maxTries
+        attempts, raise an error.
+        
+        We want to clean up after ourselves. But some filesystem backends (e.g.
+        NFS, GPFS) either create new files in the hierarchy when deleted files
+        are still open ("silly rename"), or else sometimes can't keep up with
+        filesystem operations and lack perfect read-your-own-writes
+        consistency. This code works around that. It can be deleted if and when
+        we check the mount type of the filesystem we are using to make sure it
+        is not flaky/fake.
+        
+        """
+        
+        for tryNumber in range(maxTries):
+                # Try a few times to delete everything
+        
+            try:
+            
+                for baseName in os.listdir(path):
+                    # For each file in this directory
+                    fullName = os.path.join(path, baseName)
+            
+                    if os.path.isdir(fullName):
+                        # Recurse into directories
+                        self._recursiveDelete(fullName, maxTries)
+                    else:
+                        try:
+                            # Try to delete files
+                            rm_f(fullName)
+                        except OSError:
+                             if baseName.startswith(".nfs"):
+                                # If we can't delete NFS files, skip them for now
+                                if not self._warnedOnNfs:
+                                    # But warn the user they exist
+                                    self._warnedOnNfs = True
+                                    logger.warning("Open NFS files like {} may prevent job or job store deletion".format(fullName))
+                                # Note that if the file deosn't go away soon we will fail deleting the parent directory.
+                                continue
+                             else:
+                                raise
+                
+                # Now we have deleted all the files in the directory (or skipped out on NFS files).
+                # Try to remove the directory.
+                os.rmdir(path)
+                
+                # If it worked, we are done
+                return
+            except:
+                # Something went wrong
+                if tryNumber + 1 >= maxTries:
+                    # This was our last attempt
+                    raise
+                else:
+                    # Wait and try again
+                    timeToSleep = 2 ** tryNumber
+                    logger.warning("Cannot delete {}; retrying in {} seconds".format(path, timeToSleep))
+                    time.sleep(timeToSleep)
 
     def destroy(self):
         if os.path.exists(self.jobStoreDir):
-            shutil.rmtree(self.jobStoreDir)
+            self._recursiveDelete(self.jobStoreDir)
 
     ##########################################
     # The following methods deal with creating/loading/updating/writing/checking for the
@@ -165,7 +236,7 @@ def delete(self, jobStoreID):
         # The jobStoreID is the relative path to the directory containing the job,
         # removing this directory deletes the job.
         if self.exists(jobStoreID):
-            shutil.rmtree(self._getAbsPath(jobStoreID))
+            self._recursiveDelete(self._getAbsPath(jobStoreID))
 
     def jobs(self):
         # Walk through list of temporary directories searching for jobs


Test patch: