Repository: DataBiosphere/toil

Issue:
Worker reports job success even when cache clean-up fails or never happens
Issue #963 brought up the issue where we were seeing jobs attempting to delete global files in use by other jobs (which shouldn't have been happening). @hannes-ucsc and I pinpointed it to being due to jobs saying they're completed and then being killed by bad worker before the job's temp dir was cleaned up. This left behind stale hard links that skew everything. Jobs should not be allowed to start the async update till their temp dirs have been cleaned up of all hard links.
I managed to reproduce teh bug in PR #972.

This Issue proposes to refactor Job._execute into 2 functions (names can be changed), 
1. Job._executor - A context manager that sets up stats and logging before the yield, and finishes up the stats and logging, and async updates the job after the yield
2. Job._runner - Actually runs the job, and serialises the next job(s)

worker.py will be modified to now have a nested context manager of Job._execute on the outside and job.fileStore on the inside such that the async update happens only after the cleanup has completed.

@benedictpaten  could you give us your comments on this when you can?

Worker reports job success even when cache clean-up fails or never happens
Issue #963 brought up the issue where we were seeing jobs attempting to delete global files in use by other jobs (which shouldn't have been happening). @hannes-ucsc and I pinpointed it to being due to jobs saying they're completed and then being killed by bad worker before the job's temp dir was cleaned up. This left behind stale hard links that skew everything. Jobs should not be allowed to start the async update till their temp dirs have been cleaned up of all hard links.
I managed to reproduce teh bug in PR #972.

This Issue proposes to refactor Job._execute into 2 functions (names can be changed), 
1. Job._executor - A context manager that sets up stats and logging before the yield, and finishes up the stats and logging, and async updates the job after the yield
2. Job._runner - Actually runs the job, and serialises the next job(s)

worker.py will be modified to now have a nested context manager of Job._execute on the outside and job.fileStore on the inside such that the async update happens only after the cleanup has completed.

@benedictpaten  could you give us your comments on this when you can?



Discussion:

Resolving patch:
diff --git a/src/toil/job.py b/src/toil/job.py
--- a/src/toil/job.py
+++ b/src/toil/job.py
@@ -1372,7 +1372,7 @@ def deleteGlobalFile(self, fileStoreID):
             """
             with self._CacheState.open(self) as cacheInfo:
                 jobState = self._JobState(cacheInfo.jobState[self.hashedJobCommand])
-            if fileStoreID in jobState.jobSpecificFiles.keys():
+            if jobState.isPopulated() and fileStoreID in jobState.jobSpecificFiles.keys():
                 # Use deleteLocalFile in the backend to delete the local copy of the file.
                 self.deleteLocalFile(fileStoreID)
                 # At this point, the local file has been deleted, and possibly the cached copy. If
@@ -1479,7 +1479,7 @@ def _createCacheLockFile(self, tempCacheDir):
                 'cached': 0,
                 'sigmaJob': 0,
                 'cacheDir': self.localCacheDir,
-                'jobState': collections.defaultdict(int)})
+                'jobState': collections.defaultdict(dict)})
             cacheInfo.write(personalCacheStateFile)
 
         def encodedFileID(self, JobStoreFileID):
@@ -1927,6 +1927,9 @@ def updateJobReqs(self, fileSize, actions):
                 # and it is added if the file was removed form the cache.
                 self.jobReqs -= (fileSize * multiplier)
 
+            def isPopulated(self):
+                return self.__dict__ != {}
+
         @staticmethod
         def _pidExists(pid):
             """
@@ -2440,18 +2443,20 @@ def _serialiseExistingJob(self, jobWrapper, jobStore, returnValues):
     def _run(self, jobWrapper, fileStore):
         return self.run(fileStore)
 
-    def _execute(self, jobWrapper, stats, localTempDir, jobStore, fileStore):
+    @contextmanager
+    def _executor(self, jobWrapper, stats, fileStore):
         """
-        This is the core method for running the job within a worker.
+        This is the core wrapping method for running the job within a worker.  It sets up the stats
+        and logging before yielding. After completion of the body, the function will finish up the
+        stats and logging, and starts the async update process for the job.
         """
-        if stats != None:
+        if stats is not None:
             startTime = time.time()
             startClock = getTotalCpuTime()
         baseDir = os.getcwd()
-        #Run the job
-        returnValues = self._run(jobWrapper, fileStore)
-        #Serialize the new jobs defined by the run method to the jobStore
-        self._serialiseExistingJob(jobWrapper, jobStore, returnValues)
+
+        yield
+
         # If the job is not a checkpoint job, add the promise files to delete
         # to the list of jobStoreFileIDs to delete
         if not self.checkpoint:
@@ -2461,13 +2466,13 @@ def _execute(self, jobWrapper, stats, localTempDir, jobStore, fileStore):
             # Else copy them to the job wrapper to delete later
             jobWrapper.checkpointFilesToDelete = list(Promise.filesToDelete)
         Promise.filesToDelete.clear()
-        #Now indicate the asynchronous update of the job can happen
+        # Now indicate the asynchronous update of the job can happen
         fileStore._updateJobWhenDone()
-        #Change dir back to cwd dir, if changed by job (this is a safety issue)
+        # Change dir back to cwd dir, if changed by job (this is a safety issue)
         if os.getcwd() != baseDir:
             os.chdir(baseDir)
-        #Finish up the stats
-        if stats != None:
+        # Finish up the stats
+        if stats is not None:
             totalCpuTime, totalMemoryUsage = getTotalCpuTimeAndMemoryUsage()
             stats.jobs.append(
                 Expando(
@@ -2478,6 +2483,20 @@ def _execute(self, jobWrapper, stats, localTempDir, jobStore, fileStore):
                 )
             )
 
+    def _runner(self, jobWrapper, jobStore, fileStore):
+        """
+        This method actually runs the job, and serialises the next jobs.
+
+        :param class jobWrapper: Instance of a job wrapper object
+        :param class jobStore: Instance of the job store
+        :param class fileStore: Instance of Job.FileStore or Job.CachedFileStore
+        :return:
+        """
+        # Run the job
+        returnValues = self._run(jobWrapper, fileStore)
+        # Serialize the new jobs defined by the run method to the jobStore
+        self._serialiseExistingJob(jobWrapper, jobStore, returnValues)
+
     def _jobName(self):
         """
         :rtype : string, used as identifier of the job class in the stats report.
diff --git a/src/toil/worker.py b/src/toil/worker.py
--- a/src/toil/worker.py
+++ b/src/toil/worker.py
@@ -328,15 +328,14 @@ def recursiveDelete(jobWrapper2):
 
                 # Create a fileStore object for the job
                 fileStore = FileStore(jobStore, jobWrapper, localWorkerTempDir, blockFn)
-                with fileStore.open(job):
-                    #Get the next block function and list that will contain any messages
-                    blockFn = fileStore._blockFn
-
-                    job._execute(jobWrapper=jobWrapper,
-                                 stats=statsDict if config.stats else None,
-                                 localTempDir=fileStore.localTempDir,
-                                 jobStore=jobStore,
-                                 fileStore=fileStore)
+                with job._executor(jobWrapper=jobWrapper,
+                                   stats=statsDict if config.stats else None,
+                                   fileStore=fileStore):
+                    with fileStore.open(job):
+                        # Get the next block function and list that will contain any messages
+                        blockFn = fileStore._blockFn
+
+                        job._runner(jobWrapper=jobWrapper, jobStore=jobStore, fileStore=fileStore)
 
                 # Accumulate messages from this job & any subsequent chained jobs
                 statsDict.workers.logsToMaster += fileStore.loggingMessages


Test patch: